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ABSTRACT 

The research project investigated whether expert 
system tools have become sophisticated enough to be applied 
efficiently to problems in special education. (Expert systems are a 
development of artificial intelligence that combines the computer's 
capacity for storing specialized knowledge with a general set of 
rules intended to replicate the decision-making process of a human 
expert.) To assess the feasibility of the technology, a series of 
prototypes was developed, in which a range of expert system 
development software tools and hardware systems was used. These 
prototype systems, which sampled administrative assessment and 
instructional problems, addressed: (1) classification of students as 
learning disabled; (2) classification of students as behaviorally 
disturbed; (3) classification of students as intellectually 
handicapped; (4) classification of students as having articulation 
problems requiring special education; (5) advic-s to teachers planning 
specific procedures to deal with behavior problems (Behavior 
Consultant); and (6) development of a second opinion of the 
appropriateness of the decision-making process used in the 
development of Individualized Education Programs (Mandate 
Consultant) . Additional data arje presented on the two prototypes that 
were taken through more extensive develoj^ent and field testing. It 
was concluded that a need for the technology exists in special 
education and that it is possible to develop practical expert systems 
with the tools and research and development resources presently 
available. Nine appendixes comprising the bulk of the document are 
concerned with expert systems in relation to such topics as: (1) 
individual education program planning; (2) diagnosing, classifying, 
and treating learning disabled students? and (3) evaluating the 
development of such systems in education. (KM) 
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Abstract 

The question of primary concern in this project was: Have 
expert system tools becomo sophisticated enough to be applied 
efficiently to proolems in special education? 

The expert system/ a *.cchnology within the field of 
artificial intul ligence, was developed in response to the need 
for expertise in areas where field problems are complex and 
expert help is limited and expen-sive. Expert systems had been 
developed in industry, defense projects, and health science. 
There had been little research done on the application of the 
technology to education, and special education in particular. 
This project represented one of the £irst efforts to assess the 
potential of expert systems to solve some of the problems in 
special education. 

To assess the feasibility of the technology, a seriet of 
prototypes were developed. These prototypes sampled 
administrative assessinent and instructional problems in special 
education. In the prototype development process, project staff 
experimented with a range of expert 3y3tem development software 
tools and hardware systems. 

Two of the prototype systejis were then taken through more 
extensive development and field testing. Extensive data on user 
reliability, decision validity, and user and administrative 
acceptability was collected. 

It was concluded that there were important problems in 
special education that could benefit from the application of 
expert systems technology. It was also concluded that it was 



possible to develop practical exportable expert systems with the 
tools presently available and with the limited research and 
development resources available to the field of special 
education. There was every indication that the cost of the 
software development tools would continue to decrease, and the 
power and flexibility of the hardware and software would continue 
to increase. 




Introduction 

During a presentation to Congress/ former Secretary of 
Education Bell (1983) made the observation that "Too much 
computer software is simply electronic page turning, and it has 
little advantage over a well-illustrated book" (p. 4). The 
Secretary then called for the application of ". . . artificial 
intelligence to interact with the minds of learners" (p. 4). 
Unlike industry, medicine, and the defense fields, public 
education has done very little research and development on the 
applications of artificial intelligence to the problems faced by 
educators. There are several reasons why educators have not been 
active in this area. 

First/ the technical and personnel resources necessary for 
the development of artificial intelligence (AI) products have, 
until recently, been rare and expensive. Second, the long-term 
efforts necessary for AI product development did not fit the 
funding patterns for educational research. in referring to 
examples of effective expert systems (one type of AI product), 
Winston (1979) noted that no one should look at these systems 
without understanding the "years of team effort that have gone 
into translating the basic strategies into working, useful 
systems" (p. 273). 

Expert Systems 

Pecently, the expert system area of AI has received a great 
deal of attention. An expert system is a computer program which 
attempts to replicate the decision-making and problem-solving 



skills of knowledgeable and effective human experts. 
Waltz (1983) reported that: 

The biggest AI news of the recent past has been the 
commercial introduction and industrial use of a number of AI 
systems, especially NL (Natural Language) and expert 
systems. This news is significant because (1) it has 
quieted critics who argued that AI would never produce 
useful results, and (2) the applications themselves have 
high intrinsic value. (p. 56) 

If expert systems technology can be applied to special 
education's problem domain, a number of benefits are likely. 
First/ the quality of the special education knowledge base might 
be improved. In addition, diagnostic and treatment resources 
could be significantly expanded. In discussing the value of AI 
products, Hayes-Roth, Waterman, and Lenat (1983) state. 

We can anticipate two beneficial effects. The first and 
most obvious will be the development of knowledge systems 
that replicate and autonomously apply human expertise. For 
these systems, knowledge engineering will pro/ide the 
technology for converting human knowledge into industrial 
power. The second benefit may be less obvious. As an 
inevitable side effect, knowledge engineering will catalyze 
a global effort to collect, codify, exchange, and exploit 
applicable forms of human knowledge. In this way, knowledge 
engineering will accelerate the development, clarification, 
and expansion of human knowledge itself, (p. xii) 
A third potential benefit of expert system development and 
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use is improved training. where expert systems have been 
employe^, users of these systems have experienced a training 
effect. Furthermore/ specific efforts to modify expert syste-ns 
for training purposes suggest that such modification can enhance 
the value of the systems. 

The knowledge generation and clarification activities 
associated with expert systems development have research 
implications of considerable value. In perhaps the only well- 
validated AI program with significant implications for special 
education, DEBUGGY, the knowledge generation aspect was 
significant. The developers. Brown and Burton (1978)/ added 
considerably to our knowledge of student errors in arithmetic. 
With DEBCGGY/ the user is trained to identify error patterns in 
students' attempts at arithmetic problems. In developing the 
program's knowledge base. Brown and Burton not only arrived at an 
estimate of the percentage of errors that were systematic (80 
percent)/ they also documented the different types of systematic 
errors and their relative frequencies. 

Expert System Development Tools 

Because of a recent software development (the expert systems 
development tools)/ research in artificial intelligence has 
ceased to be the province of a few basic researchers working in 
well-endowed lalJt>ra tories. These tools allow lesser-trained 
individuals to apply AI technology. This trend has been summed 
up as follows: 

Training in knowledge engineering usually requires several 
years of study at one of a handful of universities. A group 
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of us in the knowledge systems area at Xerox PARC is trying 
to shorten this training time. Our goal is to increase the 
impact and scale of knowledge engineering by simplifying ihe 
methods of knowledge programming and making them more widely 
accessible. (Stefik, et al./ 1983/ p. 4) 

When the proposal for this study was written, a number of 
expert systems development tools were available, and software 
houses were promising that new tools would be released. 

Problem Statement 

The problem addressed by this study was that no literature 
or data existed which described the application of expert systems 
technology to special education. We didn't know if the 
application of thia technology to special education's problems 
was feasible. 

The question of primary concern in this project was: Have 
expert ayatea tools becoae sophisticated enough to be applied 
efficiently to problens in special education? 

Implicit in this question was the notion that such 
educational applications should require fiscal and time resources 
no greater than those normally available to researchers in 
special education. 

Program Objectives 

To answer this question, a series of activities were planned 
to accomplish three key objectives. The first objective of the 
AI feasibility project was to evaluate AI expert system 
development tools. The second project objective involved 
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designing small prototype systems to evaluate the potential of 
expert system authoring tools in special education applications. 
The third objective was to develop and test a practical expert 
system to help deal with one special education area« 

Objective 1; Comparison of expert system tools . During the 
project year/ staff completed a review of a variety of expert 
system tools. As a result of this work/ three articles/ which 
compare expert system tools/ have been published. The articles 
were co -authored by project staff/ graduate students and 
colleagues from other disciplines (Lubke/ Ferrara/ & Parry/ 1985; 
Perrara/ Parry/ & Lubke/ 1985; Ludvigsen, Grenney/ Dyreson/ and 
Ferrara/ 1986). Copies of these articles are appended to this 
report. Table 1 summarizes the tools reviewed by project staff 
and their opinions/ which are detailed in the appended articles 
(see Appendices h, Bf and C). 

Objective 2; Development of prototype systems . Over the 
course of the two-year feasibility study/ project staff have 
developed prototype systems. A few of these systems were/ in 
retrospect/ ill-conceived. The majority are either being fully 
developed (with other funding) or have the potential to be 
completed . 

CLASS. LD . The earliest prototype system was CLASS.LD. The 
system was designed to give a second opinion on decisions to 
classify students as learning disabled. The first version was 
developed using Expert-Ease as the development tool (Hofmeister/ 
1983). The system was then revised/ and further development 
continued using the more flexible and powerful M.l system 
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(Perrara & Hofmeister/ 1984). CLASS. LD became the primary system 
to check the flexibility^ power/ and ease of use of different 
development tools. Appendix D (Hofmeister & Lubke, 1985) 
contains more detailed information on CLASS. LD. 

CLASS. BP and CLASS. IH . These two prototypes were designed 
to test the modular structure of interrelated programs. CLASS. BD 
was designed to give a second opinion on classification decisions 
related to behavioral disturbance^ and CLASS.IH was designed to 
provide similar information for intellectually handicapped 
classification decisions. 

These are potentially large programs/ and project resources 
did not exist to take them past their prototype forms. Work on 
CLASS. BD was recently continued as a part of a cooperative effort 
with the Utah SEA. The SEA is extremely interested in the system 
for three reasons: (1) the SEA's need to improve field decision 
making: (2) the need for an inservice training tool/ and (3) the 
expert system's knowledge base is helpiag the SEA refine state 
regulations related to the classification of students as 
behavioral ly disturbed. 

CLASS.IH has been left as a rather primitive prototype 
because of a lack of resources to continue development. 

CLiiSS.SP . A prototype system was developed to provide a 
second opinion regarding a student's articulation problems and 
the student's eligibility for special education services. This 
system was left as a prototype for lack of resources and limited 
field interest. 

Mandate Consul tan t . This expert system was designed to 
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provide a second opinion on the appropriateness of the decision- 
making procedures used in the development of an lEP. One major 
purpose of the system was the reduction in the need for hearings 
to resolve conflicts between parent and school. 

Because of the extensive interest shown in the prototype by 
SEA and LEA special education administrators/ additional 
resources/ in the form of a student initiated research grant/ 
were obtained to take the system through extensive field testing. 
This field testing has been completed (see Appendix E, Parry & 
Hofmeister/ 1986) for a summary of the field-test results. 

Behavior Consultant . The Behavior Consultant was designed 
to explore the potential of expert systems with a more 
instructional emphasis. The previous prototypes had stressed the 
assessment and administrative aspects of special education. The 
Behavior Consultant prototype was designed to give advice to 
teachers planning specific procedures to deal with behavioral 
problems. The expert system was designed for both field 
consultant and clinical training applications (see Appendix F, 
Serna,, Baer, & Perrara/ 1986). In our initial evaluation of the 
prototype/ it appeared that the size and complexity of the 
program would require a mainframe rather than a powerful 
microcomputer. In the last review it appeared that recent 
developments in software/ particularly the move from Lisp to the 
"C" language^ would allow the use of microcomputers for this 
program. The program is being viewed as (1) an instructional 
planning and management tool for the teacher/ and (2) a training 
tool for advanced practicum experiences. Development is 
continuing with this system/ using training resources. 
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Objective 3t System testing and validation . Information 
regarding the validation of expert systems in education has been 
limited prior to this study, Hofmeister (in press/ see Appendix 
G) describes a validation and expert system development model 
created for this project. This model has been employed in the 
development in a number of the prototype systems described in the 
previous section. 

The validation of an expert system is a complex process. 
There are several ways in which to evaluate a system like 
CLASS.LD. First/ there are issues of basic interuser 
reliability. Two users may/ for example/ look at the same data 
source/ e.g./ a student's file, and find different information. 
How well one knows a student may have an impact on a user's 
responses to the expert system's questions. Finally/ the setting 
in which a system is used will affect its reliability. If 
CLASS. LD is used for a "private" consultation/ one set of data 
may be provided. If the computer program is used to provide a 
second opinion at a staffing/ a different set of data describing 
the same student may be entered. Reliability is then threatened 
by a variety of factors which have little to do with the internal 
function of the machine. 

A second issue involves the validity of the system. Does 
its knowledge base accurately reflect the best thinking of 
content area experts. 

Two additional issues address the system's impact on its 
environment. If a reliable and valid expert system is not used 
or if its advice is ignored/ the value is limited. A system's 
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values can be measured in a variety of implementation situations* 
CLASS.LD2 could be used in training^ field consultant/ and 
program auditing roles. 

This study's validation efforts addressed two issues with 
regard to CLASS. LD2. First/ the degree to which initial data 
suggests agreement with content area experts was measured. 
Second/ the efficiency of CLASS. LD2 in a training situation was 
evaluated. Two articles describe this project's findings 
recognizing these issues. The article by Martindale/ Ferrara/ 
and Campbell (submitted for publication) describes the validity 
study. The article by Ferrara/ Prater/ and Baer (in press) 
provides preliminary information regarding CLASS. LD2's ability to 
function in two different training situations. Both articles are 
appended to this report. 

Summary 

Development Software 

The introduction of expert system technology to special 
education is clearly feasible. 

Too ls. Early PC-based tools were/ at best/ crude 
instruments for building simple and costly systems. When this 
project began/ mainframe tools were slow/ cumbersome monsters 
which were known to eat a VAX for breakfast. Other large expert 
system tools required $150/000 dedicated Lisp machines. In the 
two-year project period/ this somewhat grim state of affairs has 
changed dramatically. Current PC systems are fast/ efficient/ 
and capable of producing professional expert systems which 
address substantive problems. Mainframe systems are much faster 
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and require much less RAM than their predecessors. The Lisp 
version of S.l had to drag along five megabytes of Franz Lisp 
baggage. The newest version of the same shell is written in "c." 
As a result/ other users of our VAX are less likely to send us 
electronic hate mail when we log on. 

Every month has seen the introduction of new shells and more 
powerful versions of existing systems. m addition/ costs have 
gone down. We are paying less for more powerful software. 

Special Education Prototypes 

The prototype systems developed by this project suggest that 
there is no reason to delay the development/ testing/ and 
implementation of expert systems for special education. Existing 
tools have all the power we need to do a first-class job in the 
area of diagnosis/ planning/ and instruction. As tools become 
more powerful/ we can add more "bells and whistles/" but solid/ 
workable systems are certainly possible today at a reasonable 
cost. It should be kept in mind that most oi our prototype 
systems were developed by special educators with little or no 
computer programming background. Although/ to date, only two of 
these prototypes (Mandate Consultant and CLASS. LD2) are being 
fully developed/ the success of these prototypes suggests that 
complete systems are possible in each of the prototype areas. 

CLASS. LD2 . CLASS. LD2 has been demonstrated to be a valid/ 
usable expert system. If implemented/ the system may be the 
answer to some of the current overclassif ication problems in the 
area of LD classification. To suggest/ however/ that CLASS. LD2 
is the solution to anything is premature and goes well beyond the 
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data. Current SEP-funded research, as well as testing efforts 
being made by individual state education agencies, is increasing 
our certainty regarding the efficiency of CLASS. LD2 in both field 
consultant and training roles. 

Potential Problems 

There are several potential problems which may delay or 
restrict the general distribution and use of expert systems. 
First, schools may view hardware costs as prohibitive. Even our 
smal lest prototypes wil 1 NOT run an Apple lie with 64k of RAM. A 
minimally configured system requires an IBM-PC compatible 
computer, with at least 512k of memory, and two disc drives. 
Such machines are not commonly found in public school classrooms. 
They are, however, becoming common in administrative settings in 
public schools. 

Second, software licensing fees are very high by educators' 
standards. TEKNOWLEDGE, for example, charges $250 per disc for 
distribution copies of programs that include their software 
tools. This high cost may limit expert systems building efforts 
to areas where a substantial economic benefit is immediately 
apparent (as with CLASS. LD2 and Mandate Consultant). Behavior 
Consultant, and other systems designed to directly benefit 
teachers and students, may have to wait for costs to go down. 
There can be little doubt that costs will continue to drop. 

Conclusions 

Project Objectives 

All project objectives have been met or exceeded. We were 



fortunate to acquire additional university funds to supplement 
the federal resources for the project. As a result: 

1. We acquired and field tested more development tools 
than originally planned and budgeted for; 

2. More prototypes were developed than planned; 

3. More of the prototypes were taken to a field-test level 
than planned/ and 

4. More refereed publications were prepared than the 
project objectives called for. 

In addition to meeting the formally stated project 
objectives/ a number of important secondary objectives were met, 
including: 

1. Three doctoral students in special education received 
extensive internshp experiences; 

2. The Department of Special Education added important 
advanced clinical training experiences to their 
training program through the incorporation of CLASS.LD. 

3. The project was largely responsible for three SEA's and 
two other uni versi ties making investments in the 
development of other expert systems in special 
education. 

4. The project was largely responsible for an increase in 
university commitment to further research and 
development in the application of technology to the 
needs of tha handicapped. 



Feasibility of Expert Systems 
in Special Education 

We entered the project with considerable uncertainty 
regarding the practical value of expert systems in special 
education. We finished the project certain that expert systems 
can make a contribution. 

Our certainty is the result of the following observations. 

1. During the project/ two major advances in software and 
one major advance in hardware allowed expert systems of 
a practical size to be developed and run on powerful, 
yet modestly priced/ microcomputers. If we had been 
restricted to mainframe applications/ the technology 
would have been too costly and inflexible for many 
field applications in special education. 

2. After experimenting with a range of prototypes which 
sampled administrative/ assessment/ and instructional 
problems in special education/ we collected sufficient 
information to suggest that the technology could be 
adapted to selected/ important problems in special 
education. 

3. After taking two of the prototypes and developing them 
to a point where we had extensive data on product 
validity/ user acceptance/ and administrative support 
for their use/ we became convinced of the potential 
effectiveness and acceptability of expert systems in 
special education. The fact that four SEAs are now 
investing their resources in the further development 
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and adaption of these expert systems provides 
substantive evidence of field acceptance. 
The observation that some of the secondary benefits of 
expert systems development were greater than originally 
hypothesized/ provided additional data to support the 
value of the technology. One secondary benefit related 
to the value associated with the building of a 
knowledge base for an expert system. To build a 
knowledge base for an expert system/ you are required 
to collect existing knowledge in a specific area and 
organize that knowledge so that it can be aplied to the 
solution of pressing field problems. It is the 
analysis and synthesis of existing knowledge so that it 
rationally directs field decision making that allows 
the development of expert systems to make a substantive 
contribution to the need to translate our research 
findings into practice. 

Perhaps the major secondary benefit related to the 
training value of expert systems. The original intent 
of an expert system was the emulation of the field 
consultant. An expert system (1) models the decision- 
making processes of an expert/ and (2) makes its 
decision-making process aad the associated rationale 
overt. This modeling and transparency of reasoning 
combine to form an effective training function in areas 
where training is difficult and expensive. Advanced 
clinical training is one such area, and administrative 
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decision making is another. Expert systems could 
likely be justified for their training value alone. 
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t;xpert syscems in cne Inaiviaual Laucation Piun Process 

rtul tiaisciplinary teams must aevelop inuividudl eaucationiil 
plans (iLPs) tor hanaicappea cnilaren (Eaucation tor Ali, bee. 
t-Ldj 119]). The purpose or an ILP is primarily to guiae the 
aelivery ot instructicna 1 services to a hanaicappea child 
(Dualey-Marling, 1985). The process ot oeveloping an appropriate 
instructional plan begins with collecting test ano ooserva tiona 1 
aata. This intormation is useo to aetermine each chila's current 
level ot pertormance. A planning team then proceeas to aevelop 
goals and ODjectives, whicn should match tne student's 
pertormance. a review or the research lias identitieo several 
prooleins which are associateo with this element ot the ILP 
process . 

Problems in hoving trom Lata to ODjectives 

One such proolern is relatec to the quantity ano quality ot 
iniormation cjetjcrioiny sLuoent pertormance. Thurlow & ysselayke 
(l&Vy) touna that a great oeal ot o^ta oescribing stuuent 
pertormance is collectea, but mucti or it is technically 
inaaequate ana irrelevant. For example, stuaent obsec va tionax 
data, which is coiiecteo oetcce an ILP meeting, otten rails to 
opera tional ize behavior, appropriately quantity behavior, or list 
anteceaent ana consequent events. 'Auc-se limiteu ooser vationai 
recoros have little value tor program planners. 'ihey are not 
specitic enough to airect the ae v e 1 oji'men t ot goals ana 
Objectives . 

besiaes inaae^ua te Ua lj , mu 1 1 lai sc i p 1 1 na r y teams oi ten 
collect iniormation irrtlevani. to insttucticnal planning. Uorm- 
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reierencea tests, usee trequently in puolic sctiools to evaluate 
pertocaiance# can be consiaerea insttuc tiona 1 ly iccelevsint. h 
norm-re terenceu test produces a score that rellects how an 
inaiviaual*s pertormance compares with the perlormance ot other 
inuiviuuald. Foe a test to be ins true tiona 1 1 y relevant/ an 
inaiviaual*s pertorroance must be assesfsed in aosoiute terms. 

Criterion-relerenced instruments, rather than norm- 
reierencod i ns t rumen ts# assess student perlocraance in speciiic, 
precisely delinea content areas using absolute terms (borc) & 
&all# 197&}. Since cr i terion-rererencea instruments can point 
out specilic pertormance delicits# criterion- relerenceu t3Sts 
can# theretore# be more usetul to program planners than norm- 
relerencea instruments to program planners. 

An additional problem is that many instructional planners 
have oilliculty moving Irom data collection to writing 
instructional objectives. Translating cri terion-reierencec test 
Odta into prescriptive objectives is a oiilicult task. Tne tasK# 
ues^ite its level ol aiiliculcy# is critical to appcopciati. 
program planning. a student's program plan snoulu uirecLly 
relate to nis current pertormance. 

Aucnurs ano publishers oi many en x terion-relerenceu tests 
attempt to make the joD ot translating test ua ta into 
prescriptive objectives easier by providing taolcs whicn 
cetorence speciiic objectives to t;^st periornunce. tot exa.npj.e# 
Connol ly , Na tcnman# L Pritcn^tt (i970) proviue sucn reietence 
taoies loi' the Key riutn Diagnostic Miitnmetic test. 

Q 22 

ERjC 27 



In spite ot the Key Math aevelopers' oltorts to make the 
test prescriptive/ Goodstcin/ Kahn & Cawley (197b) reportea that 
Key Matn has utility only as a preliminary screening instrument 
tor assessing areas ot strength ano weakness in general 
mathematics achievement. Goodstein/ et al. (1970) telt tnat the 
usefulness ot the Key Hatn tor aiagnosis ot mathematical 
disabilities ano the prescription ot specitic intervention 
tactics remained limited. Furthermore, Goodstein et al. (1970/ 
aescribed Key Math objectives as too broad-based tor most 
teachers to aaequately develop a prescriptive program likely to 
meet individual stfident*s neeus. 

Otten skilled planners require more aetaileo and time- 
consuming, cri terion-ref erencea test data as well as additional 
information to write suitable objectives (Colburn & McLeod/ 
19&3). Many times, unskilled planners don*t even Know when to 
ask tor additional information. 

Although academic objectives are an important part ot most 
lEPs, social sKills must also oe considered. For an Ibt to be 
appropriate, objectives whicn relate to social skills must also 
be tied to a student's performance. This means that planners 
roust translate observational ua ta into objectives tor 
social/emotional oehavioral prescriptions. liince acceptable dno 
unacceptable student behavior in tlie classroom otten covers a 
uucn greater range ot circumstances than those in an acauemic 
area, the problems associateo with socia i/emotional elements oi 
program planning can become very in trices to. 
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•I'here are at least two issues that limit the liKelihooa that 
planners will write appropriate instructional oDjectives in both 
acaaemic social/emotional areas. First# inappropriate aata are 
otten collectea. Seconds planners otten lack expertise that 
allows them to translate gooa data into prescriptive objectives, 
ihese two issues are interrelated because persons untamiliar with 
handling aata appropriately cannot request aoequate intormation. 
Planners neeu aoequate intormation to write appropriate 
objectives. Without this information, implementation ot the ItP 
is severely hamperea. Failing to correctly implement an lEP can 
be consiaered the most critical detriment to appropriate 
programming tor a handicappeo chilo (Gerardi et al., 1984). 

Artiticiai intelligence; A Possible Solution 

The tield of Artiticiai Intelligence, specifically expert 
systems, may hold solutions for the problems ioentitiec in the 
research. 

Artificial intelligence is the part ot computer science 
concernea with oesigning intelligent computer systems; that is, 
systems tnat exhibit the characteristics we associate with 
intelligence in human oena v ior--unoers tanumy , language, 
learning, reasoning, solving problems, ana so on (barr oc 
feigenbdum, 19bl, p. 3). 

Artiticiai intelligence systems intenaea to replicate 
decision-making by knowleageaole ana experienceo humans are 
Cdlleo expert systeaic. An expert bystera is typically set up to 
engage the user in a oialogue. Vhis uialoguc, in many ways, 
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selected^ and (c) a set ot cri terion-ceterencea test items 
aesignea to obtain missing stuoenc intormation. 

The MTI asks questions to gather inlormation ana tnen 
analyzes the user^s answers by comparing them to the rules in the 
knowledge base, unen necessary # the system prints out aoditional 
criterion referenced test items to gather more specitic 
information aoout a student^s performance. 

behavior Consultant (BC) 

The behavior Consultant (bC) program applies expert system 
tecnnology to student behavior-problems in the classroom. 
Ultimately, two videodisc components will be associated with the 
LC expert system. The overall structure tor bC will include: 
(a) an initial vioeoaisc component designed to teacn eftective 
sKills for observing student oehavior; that is, to teach 
educators anu others to be tne "eyes and ears" of the system, (b) 
an expert system component designee to evaluate data trom the 
usee regarding student behavior-problems ana suggest strategies 
tor addressing tne beha vior-prooleois in the classroom, ana (c) a 
second videodisc component ocsigneu to teach eftective 
implementation ot the behavior strategies recommenueo oy tne 
expert system. 

Currently, the expert system couiponent ot BC is in the 
developmental stages while the vioeooisc components are in chw 
planning stages. Tne curcent version ot LC is designee as a 
ni ic rocompu te r-ba seo systeni. because ot che complexity 
anticipated tor later versions ot the expert systein, it will 
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pdrallels the type ot conversacion a person mighc have with an 
expert consultant. The computer is programmeo to ask the user 
questions to oetail tne problem or situation (barr & Feigenbaum/ 
19&1). For example/ a well-*Known meoical system tor physicians 
IS MyciU (Davis/ Buchanan & ShortJLitte, 1975}. with MYCIN the 
user inputs data into the computer intormation on the 
characteristics ot the patient's bacterial cultures and the. 
patient's symptoms. The computer is programmeo to match the 
patient's oata with information in the progcam on the 
characteristics ol bacterial cultures and then/ based on 
programmeo logiC/ present a disease diagnosis. 

Expert Systems ana lEP Planning 

Two prototype expert systems/ Math Test Interpreter (Lubke/ 
1985) ana Behavior Consultant (Ferrara & Serna , 19b5) have been 
aeveloped to test the reasibility ot applying expert systems 
technology to the tasK oi translating test ana observational aata 
into prescriptive objectives. 

hath 'lest Interpreter (MTi) 

The Math Test Interpreter (MTI) is designed to combine 
stuoent inlormation/ results trom the Key ha tn aiagnostic 
Arithmetic Test (Connolly et al./ 197o) ana additional program 
generateo cr i ter ion-*relerencea test data to produce a 
prescription tor program planning in the area ot mathematics. 

The knowleoge base ot tlie MTI con ta ins se vera 1 componen ts : 
(a) a set ot rules to guiae the consultation/ (b) d master-set ot 
oojiBfCtives trom wnicn review ano instructional oojectives are 
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ultimately be moveu to a aiaintrame computer. This paper will 
show examples irom the current bC expert system prototype. The 
basic structure employed in the current version o£ bC will also 
be used in Iciter versions transferred to the mainlrame computer. 

Expert Systems Functioning 

Consul tationa 

both prototype systems describee above engage the user in a 
dialogue. For example/ in the case ot MTI/ the user supplies 
intormation about the student such as graoo/ oata ol examination/ 
mental ability/ past math performance/ IQ, chronological age/ 
priority ratings tor content areas/ as well as item scores on the 
Key Math test. Vihen consulting fiC the user's answers to a series 
ot questions describe the bena vior-problems/ the condition in 
which the behavior takes place/ ano the condition in which the 
teacher will attempt to modity tne problem behavioc. 

£*igures la and lb present examples ot a typical consultation 
with each ot the expert systems. 

Knowledge base Rules 

Both expert systems were written using a computer language 
chat organizes human Knowledge into a series ot rules. Inese 
rules have two components: an ''it component/ or dnteceuent 
component/ and a "then" component/ or consequent component. h«hen 
the conditions in the antecedent component oi the rule ma ten the 
conditions in the user's proolem description/ a conclusion in the 
consequent component o£ tne rule is invoked. Figure 2 present an 
"il-tnen" rule taken trom the Mil. 
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what was the student's aoe in months at the time the test was 

administered? 

>> 120. 

What was the student's grade level at the time the test was 
Administered? (Enter the score as a real number/ for example 3.5 
or 6.8.). 
>> 5.1. 

Based on your information about the student's intellectual 
functioning (IQ)/ this student would be considered: 
-normally functioning (that ia, above 75) 
-intellectually handicapped (about 55-75) 
-severely-intellectually handicapped (below 60) 
>> intellectually handicapped. 

The three basic areas covered by the Key Math test are Content/ 

Operations and Applications. 

Please rate the CONTENT area in terms of priority/ 
using a 1/ 2, 3, with a "1" being the highest priority. 

>> 2. 

Please rate the OPERATIONS area in terms o'! priori ty* 

>> 3. 

Please rate the APPLICATIONS area in terms of priority, 

>> 1. 

How much time is devoted to mathematics instruction per day for 
this student? (Please enter the average amount of time per day 
in minutes). 
>> 40. 

Figure la. Typical consultation with the Math Test Interpreter 
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What is the behavior which you wish to stop or retard? 
> > talking-in-class • 

Is there a good behavior which is incompatible with talking-in- 
class? (For example! speaking normally is incompatible with 
yelling. Working on math worksheets, on the other hand/ is NOT 
incompatible with making strange noises). 
>> no. 

How quickly must the talking«-in-class be stopped? 

1. RIGHT AHAYI This talking-in-class is an immediate 

threat to the physical well-being of someone, (e.g./ 
head-banging ) . 

2. Quickly. This talking-in-class is making my life and/or 
the lives of the other kids miserable. (e.g./ 
screaming). 

3. There is no big rush/ but I'd like to stop the talking- 

in--class as soon as I can. (e.g./ talking in class). 

4. The talking-in'-class is only an annoyance. There is no 

need for a major effort to control it. (eg./ nose«- 
picking). 

>> 3. 

What consequent do you think is maintaining the talking-in-class? 
> > teacher-attention . 

Can the teacher control the teachers-attention which appears to be 
maintaining the talking-in-class behavior? 
>> yes. 

On a scale of 1 to 50/ does the student enjoy being in the 
classroom where the talking-in-class is taking place? 

He/she finds this The classroom is among 

to be an aversive place. this child's favorite place 

1 50 

>>40. 

On a scale of 1 to 50/ does the student enjoy the activities 
taking place in the classroom while the talking-in-class is 
happening? 

He/she finds these These activities are 

activities to be aversive among this child's favorites 

1 50 

>> 40. 



Figure lb. Typical consultation with the Behavior Consultant 
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If IQ « intellectually handicapped/ and 

age « AGE 

AGE « > 13/ and . 

PAST PERFORMANCE « poor 
Then EXPECTED PROGRESS « 7.5 months 



Figure 2. An example of a rule from Hath Test Interpreter. 
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ate illustrated in Figure 3* Uhen one system seeks a value tor 
an expression within a rule/ it will tirst check to see it it 
alreaay knows the value. It it has previously askeo or interrea 
the expression's value it will be storeo in the system's global 
memory. it the system looks in the global memory ana tinds a 
value tor the expression/ it will stop looking ana use that value 
to test the rule, l£ a value for the expression is not touna in 
the global memory/ the system will then seek rules which concluae 
with a value for the expression. Ihe system will then test this 
next set ot rules to identity the value ot the expression. 
Finally/ it there are no rules which concluae with a value for 
the expression/ or it all such rules fail/ the system will ask 
the user if he/she knows the expression's value. 

Figure 4 shows how faC tests a rule/ which concludes that 
time-out is an appropriate procedure for modifying ''throwing 
objects" behavior. The steps used Dy BC in this situation are 
detailed below. 

1. bC seeKs a value tor tne expression "oaa-oena vior" in 
the global memory (the global memory contains 
information alreaay acquired by asjcing the usee 
questions ) • 

2m bince tne computer alceciay had a value lor bdo-bena vior 
storeo in the global memory it returns the value 
••throwing" tor the expression "ba a-bena v ior," Tne 
expression "b" lounc in this conaition inaicates a 
variable, 'inus/ the value ••throwing" is associatea 
with the variable ••b.*^ 



EKLC 



31 

36 



MTl ernd faC programs contain tactual ana heuristic rules. 
Factual knowledge consists o£ intormation that can be uocumenteo^ 
such as state ana tederal regulations ana proven hypotheses 
(Feigenbaura & hcCorducK/ 1983). An example ot a strictly tactual 
rule would involve the calculation ot the stuaenL^s mental age 
basea on the IQ ana the chronological age input oy the user. 

Heuristic knowledge captures the "rule ot thumb" experiences 
ot humans. In special eaucation/ such knowleage mignt come iroro 
expert diagnosticians or instructors. Reterring to the rule 
presented in Figure 2/ it may be the heuristic opinion ot several 
experts that under the circumstances described in the anteceaent 
parts ot the rule, a stuaent would likely make seven ana one-nait 
months progress. 

back Chaining 

both MTl ana EC expert systems use back chaining. Ihis 
is a problem-solving tecnnique which works backwara Irom 
hypothesizea conclusions to known facts, ihus/ the expert system 
can determine ii rules succeed or tail. for example/ when 
testing the rule stated in Figure 2/ Nil tirst seeks a value tor 
the expression tor "ly." 'ihen values tor the expressions "AGL" 
ana "PASV PERFUKhAUCL" will be sought. 'rnus/ it all the 
conaitions ot a rule are coniicmed/ the conclusion is contirmeo 
and the rule succeeds, conversely/ it any ot the conditions in a 
rule cannot be contirmeo/ the conclusion cannot be coniirmea ana 
tne rule tails. 

'ihere are tnree ways in which MVl ana Lc seek values tor tne 
expressions within rules. These systems* vaii^e-seaking behaviors 
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Next, bC seeks a value tor the expression ''speea(B)," 
that iSi a value tor the "speed at which throwing must 
be stoppea." liC tinds rules concluding with "then 
speeo(B)" and tests the tirst condition or the tirst 
rule. 

bC seeks a value tor the expression "bad-behavior" in 
the global memory. 

BC returns the value "throwing" for the expression "bao- 
behavior. " 

Because quickness does not have a value in the global 
memory the system seeks a value tor the expression 
"quicKness(B)" by asking the user a question. 
BC returns the user^s value " 1-rea 1 -f as t" tor che 
expression "quickness( B)." Because the user's val ue 
"1-real-rast" does not match the value "4-rea 1 -slow" , 
this rule tails. 

BC enters "i-real-tast" in tne global memory as the 

value tor the expression "quickness(L)." 

BC considers the next rule concluding "then spceo(b)" 

and seeks a value tor the expressions "oaa-bena v lor" 

and "quickness(B)" in tne global memory. 

BC returns "throwing" ano "1-real-tast" as the values 

tor the expressions "bau-beha v ior " ano "quicknesa(B)" 

respectively. 

liie rule concluoing "then speeo(b) « tast" succeeds^ 
'rtie condition oi the original rule stating "ana SL)eea(b) 
s last" i:;» coniirmeo. 
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13. bC s^^eKs values tor the expressions in the remaining 
conditions ol ttie rule/ that is/ tiroe-out-ra tio** ana 
*chi iQ-charac teri 3 tics.** It tinas these values either 
in the global memory/ by testing rules/ or by asking 
the user. 

14. The rule succeeds or tails depending on the outcome ot 
the exprep-^ions in the premises. 

Possible System Outcomes 

Inadequa te intorma tion . both bC ana MTI can identiiy 
situations in which the aa ca -proviuea oy the user either is 
inconsistent/ lacks validity/ or is incomplete. In situations 
where this is the case/ the system will alert the user and 
suggest that aoditional inlormation should oe obtainea. MTI 
will/ in certain cases/ print out specitic cri terion-reterenceo 
test items to be administered to the student. Two options are 
available at this point; the user may continue witn the 
consultation/ or the user may abort the consultations ana gatner 
the inlormation neeaea to make a complete oiagnosis and 
prescription. Figure 5 aescribes the output ot this section ot 
the hTl consultation process. 

Ob j e c t i v es . both mti ana BC are ae signed to print 
ob:)ectives lor ILP development. Mil presents the user with two 
types ot Objectives/ review objectives and instructional 
Objectives. Review oojectives cover those isolatea skills a 
^tuaent appears to be lacking. vhe instructional objectives 
correspono witn tha level ot the uest items that xall jt or aoove 



34 

39 



Check Global 
Memory 




If the inforfflation 
is not in the meinory 




Try Rules Concluding 
With The Expression 




Figure 3. Three ways to obtain a value for an expression. 
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What is the value of item 4? (Enter a "1" if the student 
responded correctly to this item and a "0" if he/she failed to 
respond correctly) . 
>> 1. 

What is the value of item 5? 
>> 1. 

What is the value of item 6? 
>> 1. 

What is the value of item 7? 
>> 0. 

In order to determine the appropriate prescriptive objectives 
dealing with Identification and Addition of Coins and Currency^ I 
need more information. It would be helpful if you would 
administer the following short criterion-referenced check with 
your student. 



(Prints out check-test items on the printer). 



Would you like to STOP and continue with this consultation at a 
later time or would you prefer to GO OK with the consultation 
without using the additional information:. 
>> STOP. 



Figure 5. An example of a request for additional data from 
Math Test Interpreter. 
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the student's ceiling level. A student's ceiling level occurs 
wHen ne/sne has mciae curee consecutive errors on the Key Matn 
test. Figure 6 shows the screen display o£ the type ot message 
presenteo to the user at the enc ot the consultation along with 
the appropriate oDjectives. These review ana instructional 
oojectives would be appropriate to include as snort-term 
objectives in a student's lEP. 

bC proviaes terminal objectives as well as an explanation ot 
step-by--step procedures lor achieving those objectives. When the 
entire bC system is tinally completed in 19b9# the computer will 
use an interactive vioeoaisc to teach an instructor how to 
implement the suggesteo procedures. 

Other General Features ot Expert Systems 

The M.l authoring system (Teknowleoge , 1984) was used to 
create both HTl ana bC. h.l has several features which make che 
system particularly attractive to eoucators. 

1. The "TKACE" tacility aJlows the user to monitor the 

computer logic as it attempts to provioe aovice. 

2. Tne ''vajy** lacility allows the user to question the 
program aoouc ""why" it askea a question. The macnine's 
response can be an M.l rule, an English translation ol 
an M.l rule, or a reterence to state ano/or teoeral 
law. 

3. The "bhOii** tacility allows the user to query ctie 
program at any point in the consultation regaruing its 
interiueoia te concl usions. 
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The atudent needa to review the following objectives: 

3-A The atudent will verbally atate in "cents" the value of 
a penny* 

3- B The atudent will count out up to 20 penniea and 

verbally atate the amount aa penniea. 

4- C Given a nickel and five penniea the student will pcik 

out any combination of cents up to ten cents upon 
verbal inatruction. 



The following objectivea are considered appropriate for the 

atudent'a instructional level: 

6-A The atudent will be able to match each amount with the 
correct correaponding amount written when uaing a 
dollar aign and a decimal point/ given a worksheet with 
amounta written in cent form in one column* 

6- B The atudent will be able to match the numerical values 

of money word values, auch a $«50 with fifty centa. 

The student will be able to write the following 
dictated amounta using a dollar aign and decimal 
point — $1.20i $.75i $2*68/ $*62# and $.05« 

The atudent will be able to aelect the quarter when 
directed to do so# given sets of coins containing 
penniea# nickels# dimes and quarters* 

7- B The student will be able to indicate that another name 

for quarter is 25 cents when shown a quarter. 

7-C The ?de nt will be able to identify the one coin which 
ia k }.th 25 centa when given aeta of coins containing 
pennies/ nickels# dimes and quarters* 



Figur.) 6* Out- ut of ptescripclve objectives from the Math Test 
Interpreter 
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&umaiary ana Conclusiona 

bxpert systems ana special eaucation . Recent ettorts to 
apply expert systems to the problems in special eoucation 
represent a truly oitlerent approach. Considerable research is i 
neeueo betore tirm conclusions can be reacheo regaraing the value 
ot expert systems tor hanoicappeo chiloren. There are# however^ 
some preliminary tinciings that inaicate that this line ot 
research is warrantee (Holmeister & Luoice# in press). 

1. Evaluations conducteo wi th prototypes inoica te tha t 

these systems can pettorm as well as humans in specitic 
areas. 

2. Some ot the problems taceo by special educators are 

similar to the problems taceo in other disciplines 
where expert systems have been successful. 

3. The process ol assemoling ana organizing Knowledge bases 

tor expert systems is a proauctive activity in its own 
right. The development o£ the ^'it-then" rules ot a 
knowleuge case clarities existing knowleoge ana 
iaentilies areas where knowleoge is needeo. 

In teg ra t i ng e X pe r t s ystems into the ILP process . Pa pe i 
compliance is relatively easy# that is# given the time-lactoc# 
lultilling the "letter ot the law** in writing ILPo can be 
accomplishea with little etloit. Lut# mdKin<^ a oilierence in 
the quality or a hanoicappeo chi.io's CQucation# is a challenge 
thcit involves luliilimg the "spirit oi Liie law." it il 
anticipateo that expert systems liKe ana can upgraae the 
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quality oi the proaucea tor nanaicapped cniluren. Witn 

appropriatei clearly<-sta ted objectives providers can plan aaiiy 
instruction lessons that relate airectly to the iuentilieo neeus 
ot their students. Today's handicapped ciiiluren have ''more 
rights" Dut the price they pay shoulo not be "less quality 
education" (Geraroi et al.i 1984). 
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Expert Systems 
Authoring Tools for 
the Microcomputer: 

Two Examples 



Joseph M. Ferrara, James D. Parry, 
and Margaret M. Lubke 

Artificial intelligence (Al) is one of t:,e most 
excitmg areas within the field of computer science. 
Much of the current interest in Al has been a result 

cJci of computer-based expert 

systems (Buchanan and Shortliffe, 1984; Qancev 

?98V W.-'"''' LT' and MccSduck' 
fenir!^ ;''i^ol'^"J'''°^'''' ^984; Winston and 
Pfendergast. 1984). Expert systems are computer 
programs which provide users with advice. During a 

^n.w1.?"'°" "^P**" the ulr 

answers computer-generated questions. Those an- 
swers are used to teat a series of knowledge-based 
rules. When enough information is provided to 
allow the system's advice-related rules to succeed a 
potential solution to the user's problem is provided 
by the system (Weiss and Kulikowski, 1984). 

Although expert systems have been successful in 
a variety of fields, two key factors have delayed 
their use in education (Hofmeister and Ferrara, in 
press). One factor has been a lack of equipment 
Most computers owned by public schools do not 
provide an ideal expert system authoring environ- 
ment. Expert systems have typically been devel- 
oped using mainframe computers or dedicated 
LISP machines like the XEROX 1108 or the 
Symbolics 3600. Few public schools have access to 
that type of hardware. 

Programming problems have also limited the use 
of .expert systems in education. Until recently 

PPnPr5^"^' '^l'*' """^^'y LISP and 

f KULCKj. These languages are flexible and lend 
themselves to the logical rule representation re- 
quired by expert systems. They are not, however, 
easy to learn or to use. The development of a 
usable expert system often has required years of 
work by a team of skilled programmers and 
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content-area experts (Sleeman and Brown, 1982) 
Public education has lacked the resourced which 
the development of expert systems has demanded 

«iJj"'~I*'"!"^ ""crocomputer hardware cou- 
pled with the development of microcomputer- 
tased expert system authoring tools have made the 
use of Al programs within public schools appear 
more feasible today (Hofmeister and Ferrara, in 
press). Al authoring tools allow non-LISP proeram- 
mers to build expert systems. These authoring 
tools may provide users with a unique language or 
approach to program generation In addidon 
debuffiing aids, designed to assist the programmer 
may be part of the authoring tool. 

Microcomputer-based expert system authoring 
E^wi.?i ^ ■ «^««''^e"y to train novicf 
£ .I!?? en»neers. In addition, these tools can 
^.t^^ Z produce small scale, practical, expert 
systems which may be useful in solving a variety of 
edu«tiona problems (Hofmeister and Ferrara 
1984a; Hofmeister and Ferrara. 1984b; Parry and 
Ferrara. in press). ' 

. .f^''^«^''^f <^^P°'^ Software International 

i'tKT'2 (Miller Microcomputer Services) are au- 
thoring tools designed for use with microcomput- 
ers. What follows is a discussion of two of these 
authoring tools: Expert-Ease and M. 1. This discus- 
sion IS not a formal review but rather an informal 
reflection based on the experiences of the Special 
Education Al project staff at Utah State Univer- 

SI 

Utah State University is to test the feasibility of 
using expert systems to solve immediate problems 

i"jr«?^ *'''"'"'°"-. P~totype programs in the 
areas of diagnosis, classification, program evalua- 
tion, classroom management, and videodisc control 
are currenfly in various stages of development and 
testing. 

As a result of our project we have had an 
opportunity to use Expert-Ease and M.I. Readers 
should keep in mind that the comments which 
follow are not those of Al experts, but rather 
Z^^l subjective judgments after several 
months of work with these two authorina tools. 

Computer Hardware 

iRMPr "v/''^^'"'"^''^^ ""'^ IBM-PC and 

IBM-PC XT microcomputers. Both M.I and Ex- 
pert-Ease can be used with either a monochrome or 
color monitor. M.I, however, is designed to take 
advantage of a color display, whfle Expert-Ease is 
not. /M. 7 operates under PC-DOS, and we have 
found little or no problem running M. 1 on PC-com- 
patible machines. Expert-Ease, on the other hand. 
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uses UCSD and has caused problems for us on 
several IBM-compatible machines. Both systems are 
quicker and easier to use with a hard disk machine 
(more about this later). 

Learning to Use the Tools 

Expert-Ease is simple to learn and use. One 
project staff person lightheartedly contends that it 
takes about 40 minutes to learn to use Expert-Ease 
and that within five hours you are about as 
proficient as you are likely to get. While this is 
clearly an exaggeration, it isn't far from reality. We 
believe that a person with limited computer experi- 
ence can learn the system in less than one day. 
Everything one needs to know about Expert-Ease 
is contained in the manual which accompanies the 
program. The manual provides information on 
program installation, a careful tutorial giving step- 
by-step instruction in system use, and additional 
information on knowledge organization and ad- 
V£nced (larger and more complex) applications. 

M. 1 takes longer to learn than Expert-Ease: A 
four-day workshop, designed to teach students to 
use 4f.;, is provided by TEKNOWLEDGE. In 
addition, M.I novices should plan on additional 
practice before they use M.I for serious expert 
system development. This is not to say that Af.7 is 
horribly difficult. Even staff members with limited 
computer experience were able to become com- 
petent M.I users within a reasonable amount of 
time. 

During training, TEKNOWLEDGE provides 
three volumes of well-organized materials: (a) 
Sample Knowledge Systems; (b) Training Materials 
(illustrated lecture notes); and (c) Reference Manu- 
al and User's Guide. The formal workshop instruc- 
tion, combined with the M.I program materials, 
was viewed by the project staff as effective 
training. 

In addition to learning the syntax of M.I and 
Expert-Ease, a novice user of cither system would 
benefit from training in knowledge representation 
(Hayes-Roth, Waterman, and Lenat, 1983). We 
believe that knowledge representation skills facili- 
tate the design and production of improved expert 
systems. In our judgment, these skills are not 
covered adequately by either the M. 1 or Expert- 
Ecse training packages. 

Creating a Program 

Users developing programs with Expert-Ease 
employ a four-step process: (1) identify possible 
answers; (2) identify critical attributes for use in 
discriminating between differing examples; (3) 
write questions which are designed to help the 
system assign values to critical attributes; and (4) 
enter examples of problems. Expert-Ease then 



induces a logic matrix which determines and 
controls the presentation of appropriate questions 
to the user. The user's responses to those questions 
lead through the logic m^itrix to the "expert" 
conclusions. Explanations for the conclusions 
reached by the system can be identified by 
reviewing the logic matrix generated by Expert- 
Ease. In short, once appropriate outcomes, attri- 
butes, questions, and f;xamples have been entered, 
the system does the work. 

A single Expert-Ease program can handle a 
problem with up to 30 critical attributes and 300 
examples. Larger problems can be addressed by 
chaining together several expert systems and having 
each system deal with a component of the larger 
problem. When this is done, however, the system 
must load each sub-program. Without a hard-disk 
system, this can be a long and noisy operation. 

Our staff found creating Ml programs to be 
more complex. In M.I the programmer must 
identify a system goal and then develop a set of 
rules which will allow the system to question 
potential users, make inferences based on user 
responses, and arrive at conclusions which achieve 
the system's goal. M.I is much more flexible and 
feature-laden than Expert-Ease. As a result, debug- 
ging a large M 1 system is a complex job. 

To make this job easier, M.I provides program- 
mer-friendly facilities; these include: (a) a series of 
error messages which identify syntax problems; (b) 
a command which displays a system's intermediate 
conclusions; (c) a command which allows the 
programmer to print out a history of the system's 
use of rules throughout each consultation; and (d) 
a command which provides an on-screen collection 
of useful information througltout the consultation. 

Use of a hard-disk system makes authoring M, 7 
programs faster. M.I programs are not usually 
written while an authoring tool is running. Instead, 
a text editor (we use WordStar) is used to actually 
write the program. The program is then submitted 
to M. 1 for analysis and debugging. When using a 
floppy disk system, the process of moving large 
files from WordStar to M.I and back is time 
consuming. The PC-XT's hard-disk speeds this 
process. 

Consulting an Expert System 

During a consultation with an Expert-Ease pro- 
gram, questions are displayed on the screen in the 
sequence dictated by the logic tree and the user's 
answers to the program's questions. When the 
system has collected enough information to reach a 
conclusion, advice is printed for the user. When 
Expert-Ease is unable to arrive at a conclusion, the 
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user is alerted and the program can quickly be 
changed to accommodate the new case. 

Since the user may question the program in a 
number of ways, a consultetion with an At J 
program can be more complex than an Expert-Ease 
consultation. For example, an M.J program can 
provide the user with information about the 
content-related rules which have caused specific 
questions to be asked. Furthermore, an Af. / user 
can halt a consultation and have the system display 
its intermediate conclusions. In addition, the M.J 
"trace'' command can be used to print a list of the 
rules used throughout a consultation. 

Expert Systems in Education 

In the United States, only a few Al programs 
have been designed for use in schools. Further- 
more, the educational market is still limited by 
hardware costs. PC-XTs are not commonly found 
in public school classrooms. 

Additional information on microcomputer-based 
expert systems may be obtained from : 

• TEKNOWLEDGElnc.(forA/.7) 
525 University Ave. 

Palo Alto, CA 94301-1982 
Tel. (415) 327-6600 

• Jeffrey Perrone and Assoc., Inc. (for Ex- 

pert-Ease) 
368517th Street 
San Francisco, CA 941 14 
Tel. (415) 431-9562 

• Miller Microcomputer Services (for EX- 

PERT-2) 
61 Lake Shore Road 
Natick, MA 01760 
Tel. (617) 653-6136 
We believe that current problems limiting the 
use of microcomputer-based expert systems in 
schools are not insurmountable and that within the 
next few years expert systems will play an impor- 
tant role in American education. □ 
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Forthcoming Articles 

Among the articles scheduled to be published this spring in 
Educational Technology are the following: 

• A Call for Action to Improve the Design of Micro- 
computer Instructional Courseware. 

• Design Considerations for Planning a Computer Class- 
room. 

• An Analysis of Computer Software Preferences of 
Preschool Children. 

• Interactive Video: Fifty-One Places to SUrt-An 
Annotated Bibliography. 

• Making Computers Work in the V/riting Class. 

• How Are Today's Elementary Schools Using Comput- 
ers? 

• Individualizing Learning with Computer-Based In- 
struction. 

• Improving the Meaningfulness of Interactive Dialogue 
in Computer Courseware. 

In addition to these and other precedent-setting articles, the 
magazine will publish the comments of its regular Colum- 
nists, plus reviews of new products and professional 
literature, technology news, notes on new products and 
services, and numerous other regular features. 
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Expert System Tools for Civil Engineering Applications 

Phillip J. LudWgsta, Wllllaa j. Grtnacy\ Del Dyrcson', 
and Jostph M. Fcrrara 



INTRODUCTION: 



In th« past, expert systca d«v«lopaent was a 
aonuBcntal undertaking r«s«rv«d for aajor unlvarsltlas and 
corporate giants. Fortunately advances In alcroconputers 
coupled with a more pragaatlo understanding of how expert 
systea technology can be applied have Initiated a new era 
In user developed expert systeas. Developaent tlae, that 
took aultl-aan years, now takes aonths If not weeks. 
Prograaalng, which required highly technical coaputer 
skills, can now be accoapllshed by novice prograaaers with 
the aid of software tools. Accessibility to AZ 
(artificial Intelligence) expertise, that was only 
available on the university caapus. Is now available (If 
only Indirectly) through custoaer service, support, and 
training. All these changes aanlfest In one acre factor - 
the cost. Projects that once required aajor public and/or 
private funding can now be undertaken by saall coapanles 
and even Individuals. 

Expert systea software tools have played a aajor 
role In expediting prograa developaent, however, they do 
not offer a panacea to all probleas which require 
expertise, it Is laportant to know that soae probleas 
should not and possibly can not be solved by current 
expert systea technology. Por probleas that can be solved 
with current technology, consideration aust be given ^o 
the design of the tool and how It relates to your 
particular problea. Each tool, as with any software, has 
advantages vs. llaltatlons which aust be evaluated before 
project developaent begins. The final decision on which 
tool Is "best" (Bost appropriate) Is dictated by various 
factors, such as flexibility, user support, docuaentatlon, 
and of coxirse cost. 



* Ondumto RMMfch AmMmI, PretaMr. RMpwthrdr. D.pt. of Ovil Md BariromiMntal EncioMrinf . 
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Th« purpos* of thia papar is not to andora* any 
ona particular aoftvara tool to build civil anginaaring 
axpart syatam applicationa, but rathar to aaphasiza thair 
particular advantagas as vail as limitations from a civil 
anginaaring scandpoint. Tha information prasantad is 
vintaga 1985 and ona should ba awara of currant changes in 
program faaturas and prica. cavaats asida, tha body of 
this papar is based on hands-on experience and should 
prove useful. 

Brief History: 

Expert systems tools, sometimes called authoring 
tools, or more recently shells, have c relatively short 
history. Basic research of developing a tool to aid in 
building an expert system is approximately ten to fifteen 
years old. Much of the early work was done in the field of 
medical diagnosis. From this work, a tool named MYCIN 
emerged. It used "if-then" production rules, certainty 
factors and backward-chaining inference, thus setting the 
standard which many current tools follow today (1) . 

Teknowledge (Palo Alto, CA) , probably the largest and 
oldest "expert systems company" has a corporate history of 
approximately five years. Experience in developing expert 
system tools and applying them is somewhat limited, 
especially in the field of civil engineering. Fenves, 
Maher, and sriram in their paper "Expert Systems: c.E. 
Potential" highlight future uses of axpart systems but the 
lack of current applications is apparent (2) . 

It is interesting to note the evolution of these 
tools and how recent advancements affect solving problems 
within the civil engineering domain. But first, lets take 
a look at what distinguishes civil engineering problems 
from other kinds of complex problems. 

Problem Domain: 

Civil engineering exhibits an extremely wide, as well 
as deep problem domain. The sheer diversities of 
disciplines Involved and complexities encountered are 
self-evident. Because of this, civil engineering expert 
systems and thus the tools to build them mutst be extremely 
flexible. The ideal tool for building civil engineering 
systems would allow for the following: 

. * Complex mathematical manipulations within the tool 
(Including scientific functions plus canned 
algorithms, ie. statistics, integration, etc.). 

* Various forms of knowledge representation 

(Not just "if-then" production rules) . 

* Various inference strategies 

(Not just "backware-chairing") . 

* Simple calls tq other programs or expert systems 
written in ANY progrtuDming language. 

* Natural language interfaces. 

* Unlimited degree of expert system explanation. 
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* Ext^nslv^ d«v«lopmant •nvlronaents 

(•g. Intelligent editors, debuggers, graphics, 
and help facilities) • 

Unfortunately, no tool available today allows for all of 
the above. Many optlalstlc sales and customer service 
people will tout ^we can't do that directly, but we can 
show you ways to work around It^ or "our next version Is 
slated to have that Inproveaent". Upon hearing such 
statements, beware! Given the tine and aoney one can 
"work around** or wait for anything, but the tine or aoney 
night not always be available. 

A good example of what appears to be a universal 
limitation of current tools Is the ImU^lllty to handle 
complex mathematical manipulations directly within the 
tool. The standard solution is to call (sometimes 
referred to as "hook") a module written in some common 
programming language to return the desired calculated 
data. However, you might have to call this module or 
others many times within an iterative solution process. 
This can slow execution down tremendously, in some cases, 
you can use knowledge engineering (programming) "tricks" 
for greater efficiency. The drawback to this approach is 
that your solution logic is dictated more by the tool's 
limitations than by the problem. 

Even though current tools fall short of the ideal, 
the future looks bright. Expert systems and the tools to 
build them are heavily dependant on hardware speed and 
memory. This is why we see special machines designed just 
iov AI work. Fortunately, advancements in hardware design 
are bringing tremendous computing pcwer and thus making 
meaningful expert system tools available to desk top 
computers. 

Tools: 

Seven tools are investigated (EXPERT-EASE, INSIGHT, 
M.l, RuleMaster, EXPERT, ROSIE, and S.l). They represent 
a diversity of complexity, flexibility and cost. 
EXPERT-EASE, INSIGHT, and M.l are suitable for micro 
computers while RuleMaster is a medium size tool suitable 
for super minims. EXPERT, S.l, and ROSIE can be considered 
large, main frame software. 

The following criteria are used to investigate 
Important features of each tool. 

* Approximate cost. 

* Ability to handle complex mathematics. 

« Ability to interface with other software. 

* Explanation facilities. 

* Overall friendliness. 

* Documentation . 

* User Support. 

As means of a brief summary. Table 1 compares each 
evaluated tool according to the above criteria. The 
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Table 1* Expert system tools conparlson. 
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following offers a bit aor« d«tail. 
Expcrt-Eaic: 

Exp«rt-Eaa« (Expart Softwara Intarnational Ltd) ia 
probably tha aaallast of tha toola avaluatad. It is 
dasignad to aid tha usa in quick davalopnant of small 
prototypas. This Pascal basad tool faaturas an automatic 
induction routina. Ona sats up a dacision tabla and 
Expart-Easa translatas it to Pascal coda which can only ba 
axacutad from within Expart-Easa. Tha usar raally has 
vary littla control ovar tha infaranca stratagy (dadicatad 
forward-chaining). If Expart-Easa saas fit to ask a 
cartain quast ion first, tha programmar (knowladga 
anginaar) can not gat at tha Pascal code to ovar-rida tha 
tool's dacision. 

This tool doas not handla mathematical functions nor 
will it run on savaral IBM crapatibla machines. 
Expart-Easa doas not make allowances for interfacing to 
other software, plus it dees not have any explanation 
facilities. Due to these limitations, Expert-Ease can not 
ba considered for large complex problem* which involve 
mathematics, this rules out most engineering problems. 
However, Expert-Ease could be used to develop skeletal 
logic structures involved in solving larger problems. It 
has outstanding documentation and it is easy to use. Biit 
for large engineering expert systems this tool will not 
handle the load. 

Other descriptions and evaluations of Expert-Ease are 
available (3, 4, 5). 

Insight II: 

Insight II (Level Five Research) is the newest of all 
the tools evaluated. Unfortunately, we were not able to 
receive an evaluation copy in time for this paper. So, 
attributes such as overall, friendliness and quality of 
documentation are not evaluated here. Without hands on 
experience with Insight II, little can be said about its 
limitations and shortcommings, however. Insight II appers 
to have some powerful facilities at an attractive price 
($495). 

Insight II is a Pascal basad progran which boasts a 
menu - driven development environment, links to other 
programs, confidence factors, tiered explanation, the 
ability to produce "run-only" end user versions the 
ability to address 2000 rules, the speed of a compiler 
based system, and believe it nor not, complex mathematical 
functions which are intrinsic to the system. Insight II 
is the only tool evaluated which directly incorporates 
transcendental math functions (ie. cos, sin, tan, etc.). 
For such a low price, this tool might be just the ticket, 
but just how well everything fits together remains to be 
seen. 



M.1: ^ ^ 

M.l (Teknowledge Inc.) is considered by many to ba 



ERIC 



53 

60 



EXPERT SYSTEM TOOLS 23 



Sii^i;fi^^f"*'?f*?.*'^*f^ Th« reasons 

ror this analogy is 1) its price ($5000 - down from last 

TiSoS?SLf°)S?-°" only mainfraae system tools; and 3) 
TeJcnowledge offers first class user support (trainino and 
SSSSii^r^^ • ?^?«l«d9e has gone to g?Sat ling?S tl 
cfSf??;.* P"5««fion«l piece of softwire, but just as a 
Cadillac can be inappropriate for certaiA jobs, so "n 

4«*.^Jf'^^^^*"f?" ^-2) i» « Prolog based tool which 
iSJlSnl? •"flieh-like productiS rules. Se foS of 

is :i:S?;tS*2ui%S!°^*'?.°?*^"^"5- ^^^^^^^^ chaSinI can 
M 1 iSi ? "'"^^ ^« «o»ewhat awJcward. Because 

S I interpreter, rules are acted on much slower 

SSoJ 2Siiff,?"f5 be a liS^ing 

«S:r4«J? J' to constantly "hook" out to 

iS?;jfS« SfS'o^?-^ "2."°^ complex »athe»atics ot 

%uiS»?JJ*f»,'°"*'*f5 dif«ctly from within itself. 
^Hr.w ^* provided interfaces, write 

interfacing software in assembly language or "C" roh 
Boyl) and finally li^c the whole thing'JogJther iiSh M.l 
to get a new executable version. This is not a trivial 

The explanation facilities are somewhat limited 
Why? M.l will either give J p^^raSir 
™?i ^/S^*???^?" °' which caused the 

•vSIi-J 5 P°i"^ in the run. since most 

expert systems reason within networks, it is impossible 
f? « engineer to write one single e^Janation 
SL\^Jf?^°"i«'' "^J* *'i" in contixt. Some torls 

^"tSx^^'Sfr^o:? •"•-^^ ^° establish'" 
"'^ does not have an extensive development 
nS^^^' 5*°^ have its 5wn editor to 

mSJr^?;;!;! S**?T* °5 ^^^^^ion to the knowledge base, 
one must leave M.l in order to use your own text editor or 
word processor. But M.l does have nice traSiSS anS 

" the abili?? ?S produce 

rj?2r ^^"^ 8y»tems. This feature is parti?uSrly 

«S SoJ aL^Sd'^r; ^° diseminate thSir tlrHut 

sJStSS toSl? numerious copies of the expert 

The documentation is sufficient but not impressive 

f°"J*in« example expert systems. These 
examples are nice to have, but it would be nicer if each 
M.l command was defined along with numerous examples of 
3ust how that command might be used. Included is a 
helpful section on how to develop an expert system from 
proposal to turn-key delivery. This -how to" section 
»«-f"**f?? good program development advice, unfortunately 
most of this rather large section has very little to do 
with using M.l directly. ^ t;o ao 
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Thar* in bob* quMtion whether or not M.l can be used 
to build a significant expert systen (>500 rules) . The 
answer is naybe. M.l can not address nore than 200 rules 
at one tine however, allowances are nade to "shuffle" in 
and out groups of rules as needed. Here again, the 
interpreter nature of M.l aakes this process painfully 
slow. 

M.l is a powerful but expensive tool. Conplex 
systems can be built but certain inconveniences 
(interfacing difficulties and slow execution) oust be 
considered limiting. He are anxiously awaiting version 
1.4 due out in early 1986. 

Other descriptions and evaluations of M.l are 
available (6, 7, 8). 

RuleMaster: 

RuleMaster (Radian corp.) is somewhat of an enigma. 
The tool is not just one program but rather three distinct 
entities: 1) Radial is a highly structured, pascal-like 
language; 2) Rulemaker is an induction routine which 
translates "examples" (logic tables) into Radial code 
(similar to Expert-Ease) ; and 3) the User interface, a 
sophisticated menu-driven collection of editors, tools, 
and various applications which help in the building of 
RuleMaster programs. What is puzzling about RuleMaster is 
its apparent lack of a separate control structure 
(inference engine) . It is generally accepted that a 
separate control structure is one of the things that make 
an expert system - an expert system (9) . If one works 
with RuleMaster, it becomes clear that the control 
structure is up to Rulemaker and/or the programmer 
(Knowledge engineer). This aspect of RuleMaster is truly 
a double-edge sword. On the positive side, the programmer 
is forced to structure the expert knowledge into modules 
that are easily updated by Ruleuker as well as the 
programmer. On the other side, trying to produce the 
effect of anything besides forward-chaining (ie. backward 
chaining) is practically impossible. Fortiinately, many 
engineering problems are well structured and can be solved 
via forward-chaining inference. 

RuleMaster handles complex mathematical functions by 
means of the usual "hook" to a separate progreun. what 
sets RuleMaster apart from the other tools evaluated is 
its ability to run iinder Unix or Unix»like operating 
systems, since Unix can handle multitasking I/O, 
inforriation can be easily shared between any number of 
progvams written in any language. This is important if 
on» has a large number of engineering algorithms written 
in different languages. 

The explanation facilities are good.. A nice feature 
is tiered explanation. One can keep querying the computer 
to get various levels of detail to the question - why?. 
RuleMaster does a good job at translating example 
knowledge to understandable explanation in context. The 
fact that the explanation can be put into context is not a 
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l^^li^^ f-ature. if the problem is very complex, then a 
simple explanation of the rule being queried is (►asily 
misunderstood. The user interface makes working with this 

oS? ^* int«««ce was just moving 

out of development in August 1985. At that time, there 

l[J"lSlI. ^* applications were not 

I« RuleMaster has a soft-spot, it must be considered 
the lack of comprehensive documentation. For such a 

received is the training course 
notes which they call a "system user manual", user 
support is available through training and contract 
consulting. 

^The approximate cost of RuleMaster is $10,000 - 
$15,000 for IBM - PC and AT computers and $25,000 for 
supermini 's such as a VAX. Educational discounts and 
trial period arrangements are available. 

-^<-«J?5?^*"* ^« Radian is primarily a 

scientific - engineering company and its product " 

2f ^* P'^i^fl^ly designed for those domains. It is 
« X?" do have problems, you can talk to 

tu i fngineer who knows knowledge engineering rather 
than just a knowledge engineer who is not a civil 
II!!?7Sf!f*, ^" RuleMaster is an excellent tool for 
building large systems that do not require various levels 
oz ajDstraction. 

Additional descriptive information is available from 
Radian Corporation (10). 

EXPERT: 

«.v 4 was developed at Rutgers University for use in 

their biomedicine program so it is not surprising to find 
the tool's design directed at this domain. But, just as 

** "fdicine relies on expert diagnosis so do 
many problems in engineering. For example, diagnosing 
operational problems in a "sick" wastewater treatment 
system or aiding in structural design problems. 
Surprisingly, EXPERT is a Fortran based tool. It uses 
standard productions rules to represent procedural 
knowledge and in the tradition of MYCIN, EXPERT 
infeSnce^*' c«J^ainty factors within a backward-chaining 

Even though EXPERT is written in Fortran it only 
handles the standard mathematical operations of +,-,/,*, 
and **. It also does not hook out to other software (even 
Fortran) and xuiless you are a Fortran Guru with lots of 
J^ff.^f"*' **** expect to change this. The explanation 
facilities are somewhat limited. The user can ask why?, 
unfortunately the explanation only concerns the last 
question asked by the computer and not with the context of 
the question. EXPERT does have a "trace" facility which 
allows the user to follow the program's logic. For a 
large mainframe system, EXPERT'S documentation is a meager 
40 pages. This consists of an overview, a simple 
diagnostic example, and command definitions. 
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The nain advantage of axpatt ia its ability to run on 
largar conputar systaas (oura ia running on a VAX H-780) . 
Thia allowa for larga nuahar of rulaa to ba incorporated 
and executed quickly. EXPERT ia available for prototyping 
at little or no coat, however, there ia no formal aource 
of uaer aupport. EXPERT could be cuatonized into a very 
powerful engineering tool, if one haa acceaa to a 
relatively large coaputer and ia proficient in Fortran and 
fundanental expert ayatea prograaaing. If you can not 
afford cuatoaization, then plan on uaing EXPERT to aolve 
diagnoatic probleaa which do not involve coaplex 
algorithaa . 

Other descriptiona and evaluationa of EXPERT are 
available (11, 12) . 

ROSIE: 

ROSIE (Rand Corporation) haa been deacribed ac a 
general- purpoae AI language aa well aa an expert ayatea 
building tool (13). since ROSIE ia written in the 
INTERLISP prograaaing language, it naturally picked up 
aany of INTERLISP' a featurea. Aa an expert ayatea tool, 
ROSIE uaea an Engliah-oriented ayntax in ita knowledge 
baae and input/output facilitiea. At firat glance, it 
aeeaa ROSIE haa a built in natural language interface, 
however, it aakea no atteapt to graap unraatrictad Englieh 
input, one auat learn to "talk" to ROSIE in a very 
atructured aanner which reaeablea aiaple engliah 
aentencea. It ia atill ia^reaaive to aee the uaer 
interact with ROSIE by typing in aaall reporta deacribinu 
certain aituationa rather than anawering one ouery at a 
tiae. ^ ' 

Becauae ROSIE ia a large prograa which requirea a 
large language (INTERLISP) to run, it coaaandeera 
aignificant aeaory and run-tiae. If one ia paying for 
theae aervicea, the developaent coata can be reatrictive. 
A aitigating factor ia ROSIE (VAX-VMS veraion) can be 
obtained for approxiaately $200, but be prepared to apend 
aeveral thouaand dollara for INTERLISP. 

ROSIE doea not incorporate coaplex aatheaatical 
functiona aa part of the tool nor doea it aake allowancea 
for interfacing with other aoftware. The explanation 
facilitiea are very liaited. one auat uae a "trace" or 
"acan" coaaand to indirectly find out what ia going on, 
rather than juat aiaply aaking why?. The developaent 
environaent ia alao liaited. There are no build in 
editora, aenua, or graphical aide. The docuaentation 
conaiata of three voluaea and ia aufficient to get atarted 
on aaall to aediua aize ayateaa. Little advice ia given 
on building large ayateaa via ROSIE. Finally, no foraal 
aupport ia availabla fc ROSIE. Rand Corporation doea 
conault on ROSIE but doea not aupport it in a aarketable 
way. 

Additional deacriptive inforaation ia available froa 
Rand Corporation (14, 15, 16). 
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S.1: 



8y8tei'*?L?! ^k\'"t!. ^^-J**;"** th« Cadillac of saall 
iJ™ (Teknowledge) is the Rolls Royce of the 

eating one of our ti»e-S5ared vSn-?So^2%S2 

on one hand, s.l has a sophisticated davaiarm<kn4- 

tr«inTl!;i eocunentation consists of five volunes, mostly 

sJItM? ""•^ in «>»^d«»^ to build ones ovm 

«u«««Jj^?S??'* i* outstanding user 

support, little of which cones for free, but still it i« 
nice to know its there when needed. Sise peoSle nr^hJhw 
have sore experience in building eipert lJstlSs Jh!J 
other coapany. Their products 2igS Sst^Jo?; £5? tS2^ 

S!?;rees"3t^ co%':n:a%%*'fS\h^- 

'w^f **** Reeling S.l was designed to solve verv 
25?S»i??;a*'S/f n*^ technical problLs, soalSSVJJke 
automating the nail room at the pentagon, if price is no 

??2;iw?;?v''?: iL^^*""^ s.i^iiows^ssig; "° 

riliihli to handle even engineering problems, but 
5™^®V^f'*'' yourself "do I need a pickup or a Rolls 
Royce". Needless to say, s.l is not for everyone. 

Case Study: 

University, department of civil and 
fSli5^!2J!i Engineering is pioneering the application of 

o« envi?Snmental 
modeling and hazardous waste management, one 
current study deals with the development of a 

^SfrfJf**^^"^*'*^*^^ fy**" assessing organic chemical 
nobility and degradation in order to consult on soil 
treatment options. 
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Sine* this aystui would b« us«d for d«monstration 
purposes, th« portability of a PC-bassd program appeared 
attractive. For this saae reason of denonstration, the 
ability to produce "run-only" versions of the expert 
system was considered an important factor. After some 
deliberation, M.l was ehoosen for system development 
because the potential was there for building an expert 
system with the forementioned characteristics. However, 
one must be aware of the surrounding circumstances. 
First, since we are an education institution, all of the 
evaluated tools have been aq;iired at a greatly reduced 
cost or no charge at all. Second, Utah state University 
has available considerable in-house expertise in building 
expert systems as well as software engineering in general. 
Lastly, some of the newer PC-based tools, such as Insight 
II, were not available at the onset of this project. If 
carried out today, the decision of which tool to use for 
building this demonstration expert system would probably 
be different. 

Food For Thought: 

It is of utmost importance for any civil engineer who 
wishes to build an expert system to realize that one can 
learn to be a knowledge engineer rather rapidly, in fact 
many civil engineers are already knowledge engineers 
without even knowing it. Most engineers are quite good at 
extracting complex )3iowledge and based on scientific 
assumptions, produce simplified heuristics 
(Rules-of -thumb) . Obviously, one can not be just a 
knowledge engineer and expect to become a civil engineer 
overnight. For this reason, civil engineers should 
seriously consider building their own systems with the 
help of flexible and user friendly tools before hiring 
those who are not familiar with civil engineering 
problems . 
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Expert Systems: 
Implications for the Diagnosis 
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EXPERT SYSTEMS: IMPLICATIONS FOR THE 
DIAGNOSIS AND TREATMENT OF 
LEARNING DISABILITIES 



AIm M. Hofmelster and Margaret M. Lubke 



flfnct Application of artificial inteUigence to tlie problems of education is a 
relatiirely recent endeavor. This article wiU focus on one of tlie most promising 
intdligence - «rpert ^em. technology . and some of thl 
charactnistics that make expert systems -InteUigenf . Selected present and poten- 
tial a^pUcations of expert systems to the field of learning disabilities are presented 
along with examples of specific expert systems. 



pcrtise, devebping a problem-solving framework, 
and eventually making the knowledge available 
to others through a computer-based expert 
system. The expert system usually gathers infor- 
mation from the user in a dialogue format that 
simulates a consultation with a human expert. 
Many expert systems are designed to explain their 
line of reasoning in everyday English rather than 
computer code. 
Reasoning Procedures 

The expert system's reasoning proceduies, 
sometimes refened to as the inference engine, acts 
upon the combination of user-supplied informa- 
tion and information contained in the expert 
system^ knowledge base. 

To facilitate the interaction with the inference 
engine, the knowledge base is organijad into rules, 
consisting of two components, an If component 
and a "Ihen" component. When the conditions in 



AppBcation of computer technok>gy to the field 
qI learning disabifities has taken a variety of forms, 
the most common being computer-assisted in- 
siructton (CAI), computer-managed instruction 
(CMI). and computer-assisted testing (Hofmeister, 
1984c). To a large extent these appUcattons repre- 
sent reasonably well-developed procedures that 
existed before mlcroct>mputer5, but had to wait for 
the wkle^read availability of this technology to 
achieve their present populjwity. More recently, a 
new computer technoIo>)y —the expert system^ 
has been developed. 

A flekl within artificial intelligence, expert 
systems technology is concerned with the use of 
computers to capture and disseminate human ex- 
pertise. Typically, expert systems have proven ef- 
fective in medicine, geok)gy, chemistry, engineer- 
ing, and business. However, educators have le- 
cenljy begun to show an interest in this technolo- 
gy, particulariy as it can be applied to the problems 
associated with learning disabilities. This article 
reports on present and potential applkatlons of 
expert systems technology to diagnosis and treat- 
ment of learning disabilities. 

EXPERT SYSTEMS 
Knowledge Engineering and ''Expert 
Systems" 

Knowledge engineering is the term often used 
to describe the process of capturing human ex- 
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th€ V component match the conditions In the 
utei's problem description, a conclusion In the 
•than' component of the rule is invoked. The 
following is an If-then' rule taken from MYCIN, 
a medical expert system. 

Rule 27 

If (1) the gram stain of the organism is 
gram negative, and 

(2) the morphok)gy of the organism 
is rod, and 

(3) the aerobidty of the organism is 
anaerobic 

Then There is suggestive evkience (.7) 

that the klentlty of the organism 
is bacteiokles. 

Kaowkdat-BaMd Content 

A knowledge base Is built on two types of 
knowledge: factual and heuristk:. netualknow- 
kd29 consists of Infbrmatton that can be 
documented, such as stale and federal regulattons 
and proven hypodieses (Fslgenbaum & McCor- 
duck, 19S3). HeurMfc knowkdg^. In turn, cap- 
tures the rule-of-thumb experiences of humans. 
In spedal education such knowledge might come 
from expert diagnosticians or instructors. 

Devebping a knowledge base is a major activi- 
ty of consklerable vahie. In discusring the need 
to devebp InlaUgmt tutoring systems, Sleeman 
and Brown (1982) noted that 
Much remains to be discovered and made ex- 
phdt. We hope that educattonal theorists will 
find the expkit formulatkm of tutoring, expbna- 
tton and diagnostic processes inherent in in- 
telligent tutoring systems a test bed for devebp- 

hg more pradae theories of teaching and learn- 
ing, (p. 9) 
Ratlaiialn and CoKlmloiie 

During an expert-system consuhatton, the user 
can ask why the flcpert syrtm aritt a certain ques- 
tton. The foOowing diafogue is from CLASbXD2 
(Fmara & Hofmelster, 1984). an expert system 
that will be presented in more detail in foUowing 
sedkms. The expert system asks: 
Does the chiU have a learning defkrit in one or 
more of the foUowIng areas: 
listening comprehensfon 
written expressfon 
basic reading skills 
reading comprehension 
mathematics? 



Rather than respond "yes" or •^no", the user could 
ask, Vhy* The expert system would then res- 
pond: "An answer to this question will aid in deter- 
mining if the chiM's deficit(;,> are In an area which 
qualifies the chiM as 'learning disabled* under 
federal regulattons.' 

An expert system may also Include a ^how* 
feature that provides a list of the information that 
has been obtained up to that point In the consulta- 
tion. In addltton, a •^tracing' function is often 
available to display information that documents 
the problem-solving process used In reaching a 
given conduston. 

Incomplete Information and Certainty 
Factors 

A consultation may continue even when infor- 
mation requested by the expert system is in- 
complete. When the user responds "unknown* to 
a specific questton, the program may note the 
response and continue the consultation. 

However, if the expert system determines that 
missing informatton is valuable, the certainty 
associated with any conclusions is reduced. 
Because many expert systems are used in areas 
which deal with conchistons that are rarely definite, 
Icertainty-computin^ procedures become neces- 
sary. Certainty bdm an usodXfy based on a scale 
ofO-lOC. Henceac«1aintyfoctorof30woukl in- 
dk:ate a relath^ely fow level of confidence in the 
outcome, whereas a certb^^ty factor of 80 suggests 
a relatively high confident:e level. The foUowing 
is an example of an outcome and an associated 
confidence frurtor from CLASS.LD2: "Based on 
the information provkled, this chikl can be classi- 
fied as learning disabled with a certainty factor of 

9o: 

The features described above demonstrate the 
characteristics of some expert systems. Although 
terminotogy and specific features will vary among 
systems, most contain provistons for explaining the 
infeience process used in reaching a conclusion. 

PRESENT APPUCATIONS OF EXPERT 
SYSTEMS 
Int^Ulgant Dlagnoetlc Programe 

Some of the earliest applications of artificial In- 
telligence to the fleU of education focused on 
diagnosis. Specifically, in the diagnosis of learn- 
ing problems, the approaches that have been 
deemed Intelbgenf have been concerned with ex- 
plaining why a student Is making a mistake as op- 
posed to merely identifying certain skill deflclU. 
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BUGGY. One of the first and moi . substantial 
examples of an intelligent diagnostic program is 
BUGGY (Brown & Burton, 1980). which 
diagnoses learning problems in term, of the 
underlying 1)ugs' or consistent computational er- 
rors. An example of a bug would be, "When be - 
rowing into a column whose top digit is 1, the stu- 
dent gets 10 instead of 11* (Brown & Burton, 
1984). Reporting on findings from one of their 
field tests. Brown and Burton (1984) commented. 
It Is interesting to note that 107 of the 1,325 
students tested had a bug in their borrow-from- 
aero subprocedure and missed 6 of the 15 prob- 
lems on the test because of this one underlying 
bug. The characterization given by BUGGY is 
a much fairer evaluation than scoring these 
students 60 percent correct (p. 288) 
Interactive videodisc pffoflram. Developed 
by Hofmeister (1984b), this program assesses 
beginning math skills in English or Spanish and 
is capable of diagnosing 27 common bugs. The 
microcomputer that is bnked to the videodisc 
player analyzes both the correct and the incorrect 
answers and provides a listing of mastered skills 
from a possible total of 335 skills. The program 
also identifies whk:h of the 27 common computa- 
tional enrors are present (Castmond, 1984). 
Expert systems and Leemlng Dtoabllltles 
Although the previously mentioned intelligent 
diagnostic programs have applkrations to the field 
of learning disabilities, they were not initially 
designed to replicate the expertise of an LD 
specialist Two systems specifically designed as ex- 
pert systems applicable to the diagnosis and treat- 
ment of learning disabilities include a diagnostic 
and prescriptive program (Colboum & McLeod, 
1983) and a classification program (Hofmeister, 
1984a). 

Diagnosis and prescription. Colboum and 
McLeod (1983) developed an expert system in- 
tended to serve as a consultant in the process of 
diagnosis and prescription. The system was 
designed to guide the user "through the various 
stages and levels of diagnosis, from the initial 
suspicion that a reading problem may exist 
through to the point at which sufficient informa- 
tion had been gathered to plan an appropriate 
remedial program* (p. 32). The system's effec 
tivene^ has been evaluated by comparing its 
diagnostic reports with those of human diagnosti- 
cians. In summarizing the results of this com- 
parison, Colboum and McLeod reported that. 



In general, the results of the evaluation were en- 
couraging; the expert system's diagnoses were 
accurate. Furthermore, because of the system's 
speed at analyzing error pattems, its diagnostic 
reports included more information than those 
of the human diagnosticians. This was par- 
ticularly noticeable with regard to the analysis 
of phonic skills, (p. 37) 
Classification. One of the most perplexing 
problems feidng special education program admin- 
istrators in the United States is the firequent 
misclassification of students as learning disabled. 
Thus, research findings have indicated that more 
thian half the LO student population may be 
misclassified (Shepard, Smith, & Vojir, 1983; 
\feeklyke, 1983). The major problem is one of 
overclassification. 

. lb provkie a second opinion about the accuracy 
of LD placement decisions Hofmeister (1984b) 
developed an expert system, CLASS.LD. This 
program enabled indivkluab who make diagnoses 
of leamipq disabled" to check their reasoning and 
conclusbns against dedsfon rules programmed in- 
to the computer. An updated version of this pro- 
gram, CLASS.LD2 (Fanrara & Hofmeister, 1984) 
contains over 200 If-then' rules in its knowledge 
base and produces conclusions with associated 
certainty factors. With CLASS.LD2, the user can 
obtain a printed record of the rules used by the 
cor puter program and statements about how they 
were applied in reaching the conclusk>n that a stu- 
dent was or was not learning disabled. The record 
shows wha^ questions the computer program pre- 
sented, the answers the user provided, and the 
rules the program applied to make ''judgments' 
based upon those answers. 

JTURE APPUCATIONS OF 
EXPERT SYSTEMS 

Hayes-Roth, Waterman, and Lenat (1983) sug> 
gested that in addition to diagnosis, prescriptic/n, 
and classification, expert systems may be devel- 
oped in the areas d predk:lion, interpretation, 
remediation, planning, monitor ig, and instruc- 
tion. Already, several prototype programs of this 
type are being designed by staff of the Artificial 
Intelligence Research and Devebpment Unit at 
Utah State University. These prototypes, in tum, 
are used to test the feasibility of applying expert 
systems to solve problems in special education. 
Intelligent Teat Interpretation 

One prototype expert system is the Intelligent 
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Test Interpretation which yields an individual 
prescription in mathematics. Results from the Key 
Math Diagnostic Arithmetic Test (Connolly, 
Nachtman, & Pritchett, 1976), along with demo- 
graphic data, will constitute most of the data the 
user enters. Based on this information the com* 
puter program will produce a prescription for pro- 
gram planning. 

In a study that provided an information base for 
the expert system, Hofmeister (1984a) found that 
Key Mad} scores correlated .82 widi another much 
more comprehensive criterion-referenced instru- 
ment Consequently, the knowledge base built into 
the proposed expert system makes use of rules 
based on correlations between the Key Matii in- 
strument and the more prescriptive but time- 
consuming criterion-referenced instrument. 
iDtdilgent Monitoring of Pupil Berf onuance 

^4andate Consultant is a third knowledge-based 
expert system prototype being devebped. This 
system emulates the dedston-making processes of 
a human expert familiar with federal and state 
regulations pertaining to the Educat:on of All 
Handicapped Chiklren Act. Thus, the expert sys- 
tem is capable of providing school officials and 
parent advocates with expert advice on how to 
plan and implement instructional programs. The 
advice identifies the extent to which planning and 
instructional procedures are consistent with f^eial 
and state regulations. 

Mandate Consultant hoUs potential for address- 
ing many the issues currently resolved at a due- 
process hearing level, such as categorization, ex- 
tent of services, and placement dedstons for hand- 
icapped children. At this time, the primary applica- 
tion of this expert system is in the training of ad- 
ministrators and hearing officers rather than as a 
fiekl consultant. 

Classroom Behavior Consultant 

This prototype expert system was designed to 
generate behavk)r-management advice to teachers. 
The user provides information about the type of 
problem encountered and the conditions under 
which it usually occurs. The knowledge-base rules 
are organized into three sets. The first elicits infor- 
mation from the user and clarifies the type of 
behavior problem. The second set of rules deter- 
mines the cause of the problem or the factors 
associated with maintenance of the problem. Final- 
ly, the third group of rules generates recommen- 
dations about the intervention procedures deemed 
capable of successfully treating the problem. 



In its prototype form :he Classroom Behavior 
Consultant contains appro). innately 600 rules and 
runs on a powerful microcomputer. It is anticipated 
that a larger final version will include more than 
1,(X)0 rules and require a minicomputer. 

SUMMARY AND CONCLUSIONS 

Recent efforts at applying expert systems to the 
problems encountered in the flekl of learning 
disabilities differ greatly from traditional computer 
appUcations such as CAI and CMI. Considerable 
research is needed before any firm conclusions can 
be reached regarding the value of expert systems 
for identification and treatment of learning 
disabilities. However, some preliminary findings 
indk^te that this line of research is warranted. 

1. Evaluations conducted with prototypes in- 
dicate that expert systems can perform as well as 
humans in specific areas. 

2. Some of the problems faced by special 
educators are similar to those erKountered in other 
disciplines where expert systems have proven 
successful. 

3. The process of assembling and organizing 
knowledge bases for expert systems is a prc^uc- 
tive activity in its own right. The development of 
the *lf-then' rules of a knowledge base clarifies ex- 
isting knowledge and identifies areas where 
knowledge is needed. 
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DEVELOPMENT AND VALIDATION OF AN 

EXPERT SYSTEM FOR 
SPECIAL EDUCATORS 



James [X Plury and Alan M. Hofmelster 



Abstract The authors describe the development and initial validation of a 
computer-based expert system. Mandate Consultant (Plurii^ 1985), designed to review 
the regulatory procedures for developing Individual Education Programs (lEP). The 
formative process involved three phases: (a) definition of need andproposed solu- 
tion, (b) design of a prototype, and (c) progressive refinement through field-testing 
and revision cycles. The summative component included a two«phase experimental 
design for validating the accuracy of expert system ou^mt through comparisons with 
human experts. The findings indicated that the expert system-generated conclusions 
matched the conclusions of the "better^ human experts, and were considered more 
appropriate tlian the conclusions of the majority of experts. Furdiermore. 1»linded* 
evaluators Judged the expert system-generated conclusions as being equally accept- 
able as those produced by the 1>etter^ human experts, and more acceptable than 
those of the minority of experts. 



The Individual Education Program (lEP) pro- 
cess serves as a forum during which parents and 
school officials should reach agreement on the 
content and delivery of a handicapped child's 
education. When this process fails, other parties 
intervene to mediate the disagreement. If such 
mediation is unsuccessful, the parties involved 
proceed to a hearing to resolve the issue based 
on the intent of the law. 

Unfortunately, several problems are associated 
with such hearinc:. First, hearings cause delays 
and intenruptio I . in appropriate services for han- 
dicapped students (Budoff, Orenstein, & Abram- 
son, 1981). Second, hearings are costly in terms 
of money (Hende'^n & Hage 1979) an J stress 
(Redler, 198&>. Finally, hearings do not jusurp 
equitable and effective educational decisions 
(Salend & Zirkel. 1984). 

For these reasons, disagreements between 
school officials and parents should, whenew r 
possible, be resolved prior to formal hearings 
(Belsches-Simmo ns & Lines. 1984). Towurds this 
end, school officials need ^n unbia cd. 



knowledgeable consultant to objec. ely review^ , 
devebpment of an educatk)nal pn> mi ^o- ? ' in- 
dicapped student. 

A COMPUTER-BASED EXPERI SYSTEM 

An applkration froi. ^ a relatively new field within 
artificial intelligence (AI) holds promise for ac- 
complishing this task (Waterman, 1986) In recent 
years in the fields of medicine, geology, and 
engineering, specific domains of knowledge 
possessed by humcus have been cloned in 
computer-based expert systems. Typically, these 
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systems were designed to engage the user in a 
dialogue, which in many ways paralleled the type 
of conversation a person might have with an ex- 
pert consultant. The computer was programmed 
to present the user with questions, accept the 
user^s responses, and match the responses with 
information in the program. Finally, based on the 
programmed logic, conclusions were displayed to 
the user. 

Until recently, little application of expert system 
technology has taken place in the field of educa- 
tion (Hofmeister & Ferrara, in press). However, 
the increased power and availability of computer 
hardware and the gains in artificial intelligence 
technology irj^ke devebpment of expert systems 
for educators feasible. Fox example, an A! 
Research and Development Unit was established 
in 1984 at Utah State University for the purpose 
of exploring Al applications to special education. 
Subsequently, Fenrara and Hofmeister (1984) 
developed an expert system, CLASS.LD, de- 
signed to provide a second opinion about the ac- 
curacy of placement decisions for learning disabled 
students. The expert system consists of two com- 
ponents, a knowledge base and an inference 
''engine*. The knowledge base is made up of rules 
based on research findings as well as state and 
federal regulations. The inference engine, in turn, 
guides the process of bringing the knowledge base 
to bear on the specific case being reviewed. 
CLASS.LD is currently being validated; 
preUrrJnary evaluation results suggest that the 
system can perform as well as humans in classify- 
ing handicapped students as learning disabled 
(Hofmeister & Lubke, elsewhere in this issue). 

i ne purpose of this project was to devel^ )p and 
initially validate a computer-based expert system 
designed to help spedal educators review the pro- 
cedures followed in devetoping lEPs. The project 
included both a formative and a summative 
evaluation. The former involved a three-phase 
model proposed by Hofmeister (in press) for (a) 
definition of the need and proposed solution, (b) 
design of a prototype in response to the deifini- 
tion, and (c) progres^ve refinement through field- 
testing and revision. The summative component 
involved a two-phase experimental design for 
validating the accuracy of the expert system out- 
put through comparisc.>;^ with human experts. 
Development Phnu^i 
The formative stage of the expert system 



devebpment was concerned with ider^/ nng alter- 
native procedures to replace weakn^.ses. The 
direction for such activities came from a diverse 
group of consultants, including loc^l education 
agency staff, state education agencv/ personnel, 
and university-affiliated educators. 1 hese special 
educators systematteally evaluated e^ch formative 
activity, and their input subsequently guided revi- 
stons of prodtict deAnitbn, design, and develop- 
ment Consultant input also served as a beisis for 
judging successful completion of e^ch phase. 

Much of the formative portion cf * xpert system 
devebpment and evaluation niay be cnaracterized 
as a recycling process (Hofmeister, in press) 
Markle (1%7), for example, describerl the for- 
mative process as developme*^* tfirc sue 
cessive approximations, that i&, a pieces. ' ^ i- 
sion and trial and revision and trial with 7>uch 
' dependence on art and insight by the devoioper' 
(p. 137). In the case of expert systems, the suc- 
cessive approximations involve testing and revis- 
ing the system based on feedback from a small 
set of test cases designed to encomndss ^ range 
of problems. This stage is folJc A/ed by further 
testing and revision base i i a set of actual cases 
representative of the field (Hofmeister, in press). 
When this recycling proce results in no substan- 
tive changes to the system, it is judged as being 
stable. Hence the major formative procedures may 
be considered near completion. 

With Mandate Consultant, the formative 
evaluation of the prototype was extended to a 
group of spedal education professbnab represent- 
ing a state education agency, bcal education 
agencies, and higher education. Using multiple 
combinations of diverse test nases, created by the 
devebper from actual cur jiative student files, 
reviewers of the prototype (a) read the documen- 
tatbn written to accompany Mandate Consultant, 

(b) reviewed tfie test-case cumulative file data, and 

(c) ran consultations on the test cases using the 
prototype. The feedback from these In 'Viduals 
provided the data necessary for the cyclic process 
of testing and revising subsequent versbns of the 
prototype. This cycle contini>jed until no substan- 
tive modifbations were suggested by the reviewers' 
feedback. 

VUidadon Procedures 

During valldatbn the focus shifted from product 
improvement to formal assessment of the accuracy 
of the expert system's output. The experimental 
design involved two formal evaluation phases. In 
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the ftrst phase, ^ human experts reviewed the 
data of 10 representative cumuhtive case files and 
provided conclusions regarding fadhires to imple- 
ment state and federal regulatory procedures for 
lEP devebpment In addition, the expert system 
generated conclusions about the lEP procedures 
based on the same 10 cases. In the second phase, 
three additional human experts reviewed all con- 
clusions and judged their Acceptability using a 
rating scale. These reviewers were Idouble- 
bnnded", that is, they did not know the source of 
the conclusions, including not knowing that one 
of the sources was a computer program. The 
evaluators' ratings served as a ba^ for compar- 
ing the conclusions drawn by human experts and 
the expert system. 

This type of blinded evaluation of expert- 
system,* knowledge-base performance was 
originally conducted in the medical flekl to 
evaluate the expert systems, MYCIN and ON- 
COCIN Fa^, \AAraith, Clancey, Scott, Han- 
nigan, Blum, Buchanan, & Cohen. 1979; 
Hk:kam, ShorMe, BIschoff, Schott, & Jacobs, In 
press). This two-phase design evolved from earlier 
evahiabons cf MYCIN. The evahiatk^n compared 
experts^ dedstons, in which the answer is not clear- 
ly ■righf or "wrong', with the expert system's con- 
clusions (Yu et aL, 1979). 

C pie A total cf rine human experts par- 
ticipated in the study. Tsiese "experts^ were 
selected from a list provided by the state educa- 
tion agency in Utah of special educatton ad- 
ministrators and other leaders (eg., advocates, at- 
torneys, diid university-affiliated stefQ actively in- 
volved in special education in Utah. In addition, 
staff of the state agency identified those leaders 
and administrators on the list whc^ in the staffs 
judgment, were the most quafified VxpertaT. When 
identifying the 'experts* state agency staff were in- 
structed to consider potential subjects' amount and 
diversity of experiences as wen as spedafized train- 
ing. The nine most "quaBfied* human experts were 
selected for the study. Of these, the three most 
"qualified* according to the criteria described 
earlier, were selected for the second evaluation 
phase. 

The six subjects involved in the first phase of 
the study were local education agency directors 
of special education. All six had participated as 
trainers or trainees in inservice on implementation 
of procedures governing lEP development. In ad- 
dition, two of the six were qualified as due-process 
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hearing officers. The experts reported from 7 to 
22 years of special education experience viith a 
mean of 15.3 years. 

The three addittonal human experts, serving as 
evaluators for the second phase of the study, in- 
cluded a special education director, an assistant 
superintendent, and a private special education 
consultant /advocate All three had participated 
as trainers or trainees in inservk:e on lEP develop- 
ment Furthermore, they were qualified as due- 
process hearing officers. Their years of experience 
ranged from 15 to 34 years with a mean of 24.7 
years. 

Measurement and Imrtnimentatlon. To 

conduct the product validation study, it was 
necessary to gather a set of representative special 
educatk>n test cases bom a k>cal education agen- 
cy. Special educatton administrators klentified 10 
special educators who were representative of the 
servfce-deBvery continuum provkied bcally. 
These fecial educators were asked to randomly 
choose one student bom their respective student 
populattons and photocopy selected documents 
bom the studenfs cumulative file. They were also 
asked to remove any personally klentifiable infor- 
mation bom those documents. 

The test-case data inchided 7 male and 3 female 
spedal education students ranging in age bom 5 
to 17 years. Although the majority of students 
were classified as learning disabled, the handi- 
capped conditions they presented ranged from 
miU vecific learning disabled (SLD) to severely 
intellectually handkrapped (SIH). 

The human experts of the fost phase of the 
comparison study read the 10 cumulative files and 
noted discrepancies between the procedures im- 
plemented fbr a given case (as evidenced in the 
cumulative file documentatton) and the pro- 
cedures governing lEP development. The experts 
recorded their condustons for each of the 10 cases 
on a cumulative file report form. 

Simultaneously with this activity, three special 
educators independently completed consuhattons 
with the expert system using the same 10 student 
files. The authors selected to use three special 
educators so that if any of the conclusions were 
discrepant, the conclusions generated by two of 
the three consultations would serve as the expert 
system data. The conclusions generated by the 
system were transferred to cumulative file report 
forms like those completed by the human experts. 
All the reports, both human and computer, were 
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typed and printed using word processors. 

For the second ph^se of the comparison study, 
three additional human experts independently 
reviewed and rated the cumulative file reports pro- 
duced by the experts. The evaluators read the 
same 10 cumulative files as the experts in the first 
phase. They reviewed the cumulative file reports 
written by the experts and rated each according 
to acceptability, based on a four-category rating 
scale This scale was modeled after one validated 
in the ONCOCIN evaluation (Hickam et al., in 
press) as appropriately representing experts' opi- 
nions. The scale implemented for the present 
study included: 1 • Ideal: The information sum- 
marized in the report is synonymous widi what I 
would have written; 2 - Acceptable: The informa- 
tion summarized in the report differs from what 
I would have written, but it is acceptable; 3 - Less 
than Acceptable: The information summarized in 
the report is inaccurate and /or inadequate; 
however, I would consider these deficiencies 
minor; and 4 - Unacceptable: The information 
summarized in the report is inaccurate and /or in- 
adequate, and I wouM consider these deflciendes 
major. In addition, if rating a report as less than 
acceptable or as unacceptable, evaluators iden- 
tified specific deficiencies of the report 

Data analyate. The data from the cumulative 
file reports produced by the experts were coded 
by a special education graduate student who was 
unfamiBar with the study. The coded data were 
subsequendy analyzed and tabulated to determine: 
(a) the total number of conclusions produced by 
the experts, (b) the total number of interexpert 
agreements and disagreements* and (c) the total 
number of conclusions most frequently noted by 
the experts. These tabulations and comparisons 
were used to evaluate the degree to which the ex- 
perts' conclusions matched in terms of implement- 
ing regulations governing lEP development. 

The rating data from the second phase of the 
comparative siudy were compiled to determine: 
(a) percentage of expert case reports judged to be 
acceptable or unacceptable based on the 
evaluators' ratings, (b) sets of rankings of the ex- 
perts based on the ratings by the three evaluators, 
and (c) association between the sets of rankings 
(using KendaVs coefficient of concordance). These 
tabulations and comparisons were used to 
evaluate the degree to which human expert 
evaluators judged expert system-generated con- 
clusions as being equally acceptable as the con- 



clusions of human experts. 

As a part of the validation study, the reliability 
of consultation outcomes between and within users 
of Mandate Consultant was formally assessed. 
Percentages of agreement for the conclusions 
generated by the expert system were computed. 
Data resulted from three special educators in- 
dependently running the same cases using the ex- 
pert system, as well as three spedal educators each 
running the same cases at different times. The 
percentages of agreement provided measures of in- 
teruser and intrauser reliability, respectively. 

RESULTS 

PhSMl 

The expert conclusions in the first phase of the 
comparative study were tabulated based on the 
number of condustons generated, the number of 
interexpert agreements and disagreements, and 
the number of most frequently noted condustons. 
This information, summarized in Figure 1, shows 
the variation in the number of conclusions pro- 
duced by the experts. 

Ph^n 

";he ratings by the human expert evaluators in 
the second phase of the comparative study were 
used to compute the percentage of acceptable and 
unacceptable expert reports. This information ap- 
pears graphically in Figure 2. Furthermore, these 
ratings provided a basis for rank ordering the ex- 
perts according to the number of expert case 
reports each judged to be ideal or acceptable. 
These rankings were used to compute KendalTs 
coeffkdent of concordance correlation (W) to 
describe the assodation between the three sets of 
expert rankings. The results of the computations 
appear in Table 1. 
Reliability AsseMment 

A subcomponent of the comparative study in- 
volved formal assessment of reliability. Specifically, 
to assess interuser reliability three special educators 
independently completr;d consultations for each 
case. In addition, the special educators reran a 
sampling of the cases at a later time to assess in- 
trauser reliability. The number of agreements on 
conclusions was divided by the total number of 
conclustons to produce measures of interuser and 
intrauser reliability. The interuser reliability coeffi- 
cients ranged from .67 to 1.00 u'tKi a mean of .94. 
The intrauser reliability coefficients ranged from 
.75 to 1.00 with a mean of .95. Thus, a relatively 
strong agreement was found between the different 
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users, as well as for the same user over time 
DISCUSSION 

Phase I 

The results from the first phase of the com- 
parative study demonstrated that conclusions 
generated by the oq^ert system matched 'nibstan- 
tiaDy the condusir^ns o^ human experts. However, 
the human expe ' ^havior varied greatly. In some 
instances, human - . .perts comprehensively noted 
the failures to implement special education 
regulatory procedures for lEP devebpment while 
in other instances, they noted few, if any. Thus, 
this phase revealed that the conclusions generated 
by Mandate Consultant generally matched those 
of the 'bettef human ^jcperts, while exceeding the 
conclusions drawn by t^e majority of the human 
experts. ''Better^ experts were defined as those 
producing the most conclusions that agreed with 
those of other human experts. 

Most notable among the findings related to the 
number of conclusions fjenerated by the experts 
was the iimited number conf^fusions produced 
by most human expert* r;ri ci*.. i-Ily, four human 
experts produced fewer than two conclusions per 
case. Although no standard existed regarding the 
number of conclusions for cases, the evaluators 
in the second phase of the study rated numerous 
case reports as inadequate because of the few con- 
clusions reported by the expert. 



Although two of the human experts (i«., 
Human Expert 2 and Human Expert 3) identified 
substantially more conchisions than their col- 
leagues, none made as many as Mandate Con- 
sutent. This finding supports the outcome of a 
study by Coboum (1982) who developed and 
valid^'^t'^ an expert system to assist educators in 
diagviosing reading problems. Her comparison 
between expert system-geneirated diagnosis and 
human diagnosis revealed that the expert system 
provided more detailed information than human 
diagnostic reports. Such was the case with Man- 
date Consultant in this study. It appeared that the 
extensive knowledge oase contained in the struc- 
ture of an expert system allowed it to generate 
more specific i^iformation than many human ex- 
perts typically generate. 

In addition to generating the greatest numlnsr 
of conclusions, Mandate Consultant also achieved 
the greatest number of agreements with other ex- 
perts. The authors expected the number of con* 
elusions to be related to the number of interexpert 
agreements; the significant frequency of other ex- 
perts agreeing with the conclusions of Mandate 
Consultant strengthened the case that Mandate 
Consultant provided substantial amounts of valid 
information. 

The data also slfiowed occasional disagreements 
among the expert conclusions. These disagree- 
ments appeared to result from different interpreta- 
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Expert System Tools for Civil Engineering Applications 
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and Jottph M. Ferrara 



INTRODUCTION: 



In th« past, expert systca d«v«lopaent was a 
monumental undertaking r«s«rv«d for major unlvarsltlas and 
corporate giants. Fortunately advances In microcomputers 
coupled with a more pragmatic understanding of how expert 
system technology can be applied have Initiated a new era 
In user developed expert systems. Development time, that 
took multi-man years, now takes months If not weeks. 
Programming, which required highly technical computer 
skills, can now be accomplished by novice programmers with 
the aid of software tools. Accessibility to AI 
(artificial Intelligence) expertise, that was only 
available on the university campus. Is now available (If 
only Indirectly) through customer service, support, and 
training. All these changes manifest In one more factor - 
the cost. Projects that once required major public and/or 
private funding can now be undertaken by small companies 
and even Individuals. 

Expert system software tools have played a major 
role In expediting program development, however, they do 
not offer a panacea to all problems which require 
expertise, it Is Important to know that some problems 
should not and possibly can not be solved by current 
expert system technology. Per problems that can be solved 
with current technology, consideration must be given *^^o 
the design of the tool and how It relates to your 
particular problem. Each tool, as with any software, has 
advantages vs. limitations which must be evaluated before 
project development begins. The final decision on which 
tool Is "best" (most appropriate) Is dictated by various 
factors, such as flexibility, user support, documentation, 
and of course cost. 
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Utah SUto Univ., UMC 41, Lecu, UT MS» 
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' AiMdate Raaaaich Pnfcm. Aitifidal lnt«niCHK« BaMaidi * Dov^ 
D«p«. or Spwial Bdocaliea, Utah State Uahr., UMC M, Locan, UT UStS 
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Th« purpos* of thia pap«r in not to sndora* any 
on« particular softvar* tool to build civil anginaaring 
axpart ayatam applicationa, but rathar to aaphaaiza thair 
particular advantagaa aa vail aa linitationa from a civil 
anginaaring acandpoint. Tha infoniation praaantad ia 
vintaga 1985 and ona ahould ba awara of currant changes in 
program faaturaa and prica. cavaata aaida, tha body of 
thia papar ia baaad on handa-on axparianca and ahould 
prova uaaful. 

Brief History: 

Expart ayatana tools, sonatinaa callad authoring 
toola, or Bore racantly ahalls, hava c relatively ahort 
history. Basic research of developing a tool to aid in 
building an expert system is approximately ten to fifteen 
years old. Much of the early work was done in the field of 
medical diagnosis. From this work, a tool named MYCIN 
emerged. It used "if-then" production rules, certainty 
factors and backward-chaining inference, thus setting the 
standard which many current tools follow today (1) . 

Teknowledge (Palo Alto, CA) , probably the largest and 
oldest "expert systems company" has a corporate history of 
approximately five years. Experience in developing expert 
system tools and applying them is somewhat limited, 
especially in the field of civil engineering. Fenves, 
Mahar, and sriram in thair paper "Expert Syatema: c.E. 
Potential" highlight future uaea of axpart aystems but the 
lack of current applicationa ia apparent (2) . 

It ia interesting to note the evolution of these 
tools and how recent advancements affect solving problems 
within the civil engineering domain. But first, lets take 
a look at what distinguishes civil engineering problems 
from other kinds of complex problems. 

Problem Domain: 

Civil engineering exhibits an extremely wide, as well 
as deep problem domain. The sheer diversities of 
disciplines Involved and complexities encountered are 
self-evident. Because of this, civil engineering expert 
systems and thus the tools to build them must be extremely 
flexible. The ideal tool for building civil engineering 
systems would allow for the following: 

. * Complex mathematical manipulations within the tool 
(Including scientific functions plus canned 
algorithms, ie. statistics, integration, etc.). 

* Various forms of knoiwledge representation 

(Not just "if-then" production rules) . 

* Various inference strategies 

(Not just "backwarc;:-chairing") . 

* Simple calls tq other programs or expert systems 
written in ANY progrtUDming language. 

* Natural language interfaces. 

. * Unlimited degree of expert system explanation. 
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* Ext^nsiv^ di«v«lopaant •nvironaents 

(•g. intelligent editors, debuggers, graphics, 
and help facilities) • 

Unfortunately, no tool available today allows for all of 
the above. Many optimistic sales and custoner service 
people will tout ^ve can't do that directly, but ve can 
show you ways to work around it" or "our next version is 
slated to have that inproveaent". Upon hearing such 
statements, beware! Given the tine and aoney one can 
"work around" or wait for anything, but the tine or aoney 
night not always be available. 

A good example of what appears to be a universal 
limitation of current tools is the inability to handle 
complex mathematical manipulations directly within the 
tool. The standard solution is to call (sometimes 
referred to as "hook") a module %rritten in some common 
programming language to return the desired calculated 
data. However, you might have to call this module or 
others many tixaes within an iterative solution process. 
This can slow execution down tremendously. In some cases, 
you can use knowledge engineering (programming) "tricks" 
for greater efficiency. The drawback to this approach is 
that your solution logic is dictated more by the tool's 
limitations than by the problem. 

Even though current tools fall short of the ideal, 
the future looks bright. Expert systems and the tools to 
build them are heavily dependant on hardware speed and 
memory. This is why we see special machines designed just 
for AI work. Fortunately, advancements in hardware design 
are bringing tremendous computing pcwer and thus making 
meaningful expert system tools available to desk top 
computers. 

Tools: 

Seven tools are investigated (EXPERT-EASE, INSIGHT, 
M.l, RuleMaster, EXPERT, ROSIE, and S.l). They represent 
a diversity of complexity, flexibility and cost. 
EXPERT-EASE, INSIGHT, and M.l are suitable for micro 
computers while RuleMaster is a medium size tool suitable 
for super minims. EXPERT, S.l, and ROSIE can be considered 
large, main frame software. 

The following criteria are used to investigate 
important features of each tool. 

* Approximate cost. 

* Ability to handle complex mathematics. 
Ability to interface with other software. 

* Explanation facilities. 

* Overall friendliness. 

* Documentation . 

* User Support. 

As means of a brief summary. Table 1 compares each 
evaluated tool according to the above criteria. The 
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Table 1. Expert system tools comparison. 



59 



22 



EXPERT SYSTEMS IN CIVIL ENGINEERING 



ERIC 



following offers a bit aor* detail. 
Expert-Ease: 

Exp«rt-Eaa« (Expart Software International Ltd) ia 
probably tha aaallaat of tha toola avaluatad. It ia 
daaignad to aid tha uaa in quick davalopaant of snail 
prototypaa. Thia Paacal baaad tool faaturaa an autonatic 
induction routina. Ona aata up a daciaion tabla and 
Escpart-Eaaa tranalataa it to Paacal coda which can only ba 
axacutad fron within Expart-Eaaa. Tha uaar raally haa 
vary littla control ovar tha infaranca atratagy (dadicatad 
forward-chaining). If Expart-Eaaa aaaa fit to aak a 
cartain quaation firat, tha progranaar (knowladga 
anginaar) can not gat at tha Paacal coda to ovar-rida tha 
tool 'a daciaion. 

Thia tool doaa not handla nathanatical functiona nor 
will it run on aavaral IBM conpatibla aachinaa. 
Expart-Easa doaa not aaka allo%rancaa for interfacing to 
othar software, plua it doaa not have any explanation 
facilities. Due to these linitationa, Expart-Eaaa can not 
ba conaidered for large coaplex p'robleaa which involve 
■atheaatica, thia rulaa out aost engineering probleaa. 
However, Expert-Ease could be uaed to develop akaletal 
logic atructurea involved in aolving larger probleaa. It 
haa outatanding docuaantation and it ia eaay to uaa. But 
for large engineering expert ayataaa thia tool will not 
handle the load. 

Other deacriptiona and evaluations of Expert-Ease are 
available (3, 4, 5). 

Insight II: 

Insight II (Level Five Research) is the newest of all 
the tools evaluated. Unfortunately, we were not able to 
receive an evaluation copy in tiae for this paper. So, 
attributes such as overall, friendliness and quality of 
docuaantation are not evaluated here. Without handa on 
experience with Inaight II, little can ba aaid about ita 
liaitationa and ahortcoaainga, however, Inaight II appera 
to have soae powerful facilities at an attractive price 
($495). 

Insight II is a Pascal based progran which boasts a 
aenu - driven developaent environaent, links to other 
prograas, confidence factora, tiered explanation, the 
ability to produce "run-only" and user versions the 
ability to address 2000 rules, tha apaed of a coapiler 
based systea, and believe it nor not, coaplex aatheaatical 
functions which are intrinsic to the systea. Insight II 
is the only tool evaluated which directly incorporates 
transcendental aath functions (ie. cos, sin, tan, etc.). 
For such a low price, thia tool aight be just the ticket, 
but just how well everything fits together reaains to be 
seen. 

M.1: ^ ^ w 

M.l (Taknowledge Inc.) is considered by aany to be 
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fSj Sli iJJiSL^f'*'??' J.**^*?^ Th« reasons 

v2L^ J,.f2; 2P^cJ« H P''^*'* <55000 - down froa last 
JlJf InS 5 °i $10,000); 2) it has many features that 
TlkSo2?S3„f°U!5 °" only mainfraae system tools; and 3) 

cJSSJiiSf Jf;«Lf^;'* f^*" »"PP°^ (training and 
consulting) . TeJcnowledge has gone to great lengths to 

Cadillac can be inappropriate for certaiA jobs, so can 

M.l (version 1.3) is a Prolog based tool which 
iSJS;;!" «ngli»h-like productiS "Se So« of 

:i:S?;tS'hui%Sr^"?.*'?»^"^"9- ^o"*"** cha^inS can 
M 1 "f"^* ^* soaewhat awJcward. Becluse 

f 4?" interpreter, rules are acted on much slower 
~»Pil«^ based systea. This can be a liaiting 
4oS5:?i*^^J ^^ constantly -hooki lut to 

iSl?fJef ^?fJ ^ can not handle coaplex aatheaatics or 
JS? cuiS»?JJ*?i,'°"*'*f! dif*ctly froa within itself. 
iSLSS^S ^? provided interfaces, write 

interfacing software in asseably language or "C" (Oh 
Boyl) and finally link the whole thing ?o«ther iiSh M 1 
to get a new executable version. This is ?ot a tSJial 
a%%'i;b^ni» SrJcS'^'^ are"SSt%Si^i*i„ 

SSSSi °' which caused the 

•vn!i-S 12 i P°i"t in the run. since aost 

?S rvJiS?^" *'i^i" networks, it is iapossible 

f? « engineer to write one single explanation 
SL ."^JfJiS^ir ™J* *'i" in contlxt. soae torls 

s^tix^^'sfr^o:? i:r'°" •"'-'^^ estabii.r'" 

does not have an extensive developaent 
JSJe S^IJ^ini" J**'* it. Swn editor to 

SSI aS;r^?2!;! S °I «dJition to the knowledge base, 

une aust leave M.l in order to use your own text edlt^or «t- 
word processor. But M.l does have nice t^JSinJ anS 
2S;'!^?^'**'i"*i*»»- " also has the ability " produce 
"run-only" end user systeas. This feature is particular ij 

"^^^ diseainate thSiriork Eut 
smSS tSSl! nuaerious copies of the expert 

The documentation is sufficient but not iapressive 
when one considers the cost of the tool. 205?"^^?? o? 
SLB?Sr!Il!*!JSr contains exaaple expert systeas. These 
exaaples are nice to have, but it would be nicer if each 
M.l coaaand was defined along with numerous exaaples of 
Just how that command might be used. Included is a 
helpful section on how to develop an expert system from 
proposal to turn-key delivery. This "how to" section 
^f-J^^L^f?? good program development advice, unfortunately 
most of this rather large section has very little to do 
with using M.l directly. xiiiiix. 
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Th«rtt is SOB* quMtlon whether or not M.l can be used 
to build a significant expert system (>500 rules) . The 
answer is naybe. M.l can not address nore than 200 rules 
at one tine however, allowances are nade to "shuffle" in 
and out groups of rules as needed. Here again, the 
interpreter nature of M.l aakes this process painfully 
slow. 

M.l is a powerful but expensive tool. Conplex 
systems can be built but certain inconveniences 
(interfacing difficulties and slow execution) must be 
considered limiting. He are anxiously awaiting version 
1.4 due out in early 1986. 

Other descriptions and evaluations of M.l are 
available (6, 7, 8). 

RuleMaster: 

RuleMaster (Radian corp.) is somewhat of an enigma. 
The tool is not just one program but rather three distinct 
entities: 1) Radial is a highly structured, pascal-like 
language; 2) Rulemaker is an induction routine which 
translates "examples" (logic tables) into Radial code 
(similar to Expert-Ease) ; and 3) the User interface, a 
sophisticated menu-driven collection of editors, tools, 
and various applications which help in the building of 
RuleMaster programs. What is puzzling about RuleMaster is 
its apparent lack of a separate control structure 
(inference engine) . It is generally accepted that a 
separate control structure is one of the things that make 
an expert system - an expert system (9) . If one works 
with RuleMaster, it becomes clear that the control 
structure is up to Rulemaker and/or the programmer 
(Knowledge engineer). This aspect of RuleMaster is truly 
a double-edge sword. On the positive side, the programmer 
is forced to structure the expert knowledge into modules 
that are easily updated by Ruleuker as well as the 
programmer, on the other side, trying to produce the 
effect of anything besides forward-chaining (ie. backward 
chaining) is practically impossible. Fortunately, many 
engineering problems are well structured and can be solved 
via forward-chaining inference. 

RuleMaster handles complex mathematical functions by 
means of the usual "hook" to a separate progreun. what 
sets RuleMaster apart from the other tools evaluated is 
its ability to run under Unix or unix»like operating 
systems, since Unix can handle multitasking l/o, 
inforriation can be easily shared between any number of 
progvams written in any language. This is important if 
on» has a large number of engineering algorithms written 
in different languages. 

The explanation facilities are good.. A nice feature 
is tiered explanation, one can keep querying the computer 
to get various levels of detail to the question - why?. 
RuleMaster does a good job at translating example 
knowledge to understandable explanation in context. The 
fact that the explanation can be put into context is not a 
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" problem is very complex, then a | 
°' the rule being queried is I'asily * 
SiSwI^S f^* I?' interface makes working with this 

tool very enjoyable, but, the interface was just moving 
out of development in August 1985. At that time, there 
JJSlJbll. °' ^* *PPlic«tions wer4 not 

*.».^ I« RuleMaster has a soft-spot, it must be considered 
the lack of comprehensive documentation. For such a 
SSSf i^^K^i,: received is the training course 

notes which they call a "system user manual". User 
■upport is available through training and contract 
consulting. 

approximate cost of RuleMaster is $10,000 - 
:^«:J2?«f?'^ computers and $25,000 for 

supermini 's such as a VAX. Educational discounts and 
trial period arrangements are available. 

^* ***** Radian is primarily a 
scientific - engineering company and its product 

S?if*f?Sf ^* P»^i»?^^ly designed for those domains. It is 
. ^?„r? ***** ***^« problems, you can talk to 

?hSi i,^«J"! Jmowledge engineering rather 

than just a knowledge engineer who is not a civil 

II!l?iS?!f*, ^" J^^' RuleMaster is an excellent tool for 
building large systems that do not require various levels 
oi afistraction. 

» ^* Additional descriptive information is available from 
Radian Corporation (lo). 

EXPERT: 

«.v * "JJfERT yas developed at Rutgers University for use in 
tteir biomedicine program so it is not surprising to find 
the tool's design directed at this domain. But, just as 
the field of medicine relies on expert diagnosis so do 
■any problems in engineering. Por example, diagnosing 
operational problems in a "sick" wastewater treatment 
system or aiding in structural design problems. 
Surprisingly, EXPERT is a Fortran based tool. It uses 
standard productions rules to represent procedural 
knowledge and in the tradition of MYCIN, EXPERT 
infeSnce**' certainty factors within a backward-chaining 

Even though EXPERT is written in Fortran it only 
handles the standard mathematical operations of +,-,/,*, 

^*.*^*o not hook out to other software (even 
Fortran) and xuiless you are a Fortran Guru with lots of 

**** "°* •J'P'ct to change this. The explanation 
facilities are somewhat limited. The user can ask why?, 
unfortunately the explanation only concerns the last 
question asked by the computer and not with the context of 
the question. EXPERT does have a "trace" facility which 
allows the user to follow the program's logic. For a 
large mainframe system, EXPERT'S documentation is a meager 
40 pages. This consists of an overview, a simple 
diagnostic example, and command definitions. 
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S.I: 

•vstM^SLfl "i^,"/? con«i<J«r«d tha Cadillac or .aall 

t:ir»±«7JJi fof"»«ntation consists of five volumes, mostly 
S^SrS ?f material or sample expert systems. Unf ortSnatelv 
SJJ^BiiS naster index and just like M.l, no indiviSuSl 
examples ot each command are given. Here Aaain one 

ti^ltn cS™2^ir ^y?^*" unSSrStSS Why aSd"SSS 

syStSi? in order to build oneJ own 

sunnoS'^fYi;?'* S^'if" **«i' outstanding user 

SiSS^'lciiS J;-*'5k''*'^*=\'=*'»*» '^''^ but still ifs 

nice to ]cnow its there when needed. These ceoole nroh^hinj^ 

experience in building expert SJstSsShSn aSS^ 

^SS^^*"**"^*"^- '^^^^'^ products Sight cost more? bu? thS^ 

?Sl?Il"?SJe22SSS? ""^^^^ ""'^^ compensate' fSr\SS" 

grand?SL^SSl JS? ^SS'SIc^ic^rp^SiSSf SSmSJilSg-JS^. 

SSJ-I??^^"? ^^'^ 7*^^ *^ pSntagonf ' ?? prScS'is 

fluent in Lisp, s.l^llows eSough 
to handle even engineering problems, but 

yourself "do I need a pickup or a Rolls 
Royce". Needless to say, S.I is not for everyone. 

Case Study: 

Envir«^™-«frf^2 University, department of Civil and 
f2^«^^^2Jf^ Engineering is pioneering the application of 

o' environmental 
modeling and hazardous waste management. one 
current study deals with the development of a 

sytem for assessing organic chemical 
nobility and degradation in order to consult on soil 
treatment options. 
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Th« main advantage of axpatt is its ability to run on 
largar computar syataaa (ours is running on a VAX 11-780) . 
This allows for larga number of rules to be incorporated 
and executed quickly. EXPERT is available for prototyping 
*t little or no coat, however, there is no formal source 
of user support. EXPERT could be customized into a very 
powerful engineering tool, if one has access to a 
relatively large computer and is proficient in Fortran and 
fundamental expert system programming. If you can not 
afford customization, then plan on using EXPERT to solve 
diagnostic problems which do not involve complex 
algorithms . 

Other descriptions and evaluations of EXPERT are 
available (11, 12) . 

ROSIE: 

ROSZE (Rand Corporation) has been described ae a 
general- purpose AI language as well as an expert system 
*>"ilding tool (13) . since ROSIE is written in the 
INTERI.ISP programming language, it naturally picked up 
many of iNTERLlSP's features. As an expert system tool, 
ROSZE uses an English-oriented syntax in its knowledge 
base and input/output facilities. At first glance, it 
seems ROSIE has a built in natural language interface, 
however, it makes no attempt to grasp unrestricted English 
input. One must learn to "talk" to ROSIE in a very 
structured manner which resembles simple english 
sentences, it is still impressive to see the user 
interact with ROSIE by typing in small reports describing 
certain situations rather than answering one query at a 
time. ^ ■* 

Because ROSIE is a large program which requires a 
large language (INTERLISP) to run. It commandeers 
significant memory and run-time. If one is paying for 
these services, the development costs can be restrictive. 
A mitigating factor is ROSIE (VAX-VKS version) can be 
obtained for approximately $200, but be prepared to spend 
several thousand dollars for INTERLISP. 

ROSIE does not incorporate complex mathematical 
functions as part of the tool nor does it make allowances 
for Interfacing with other software. The explanation 
facilities are very limited. One must use a "trace" or 
"scan" command to Indirectly find out what is going on, 
rather than just simply asking why?. The development 
environment is also limited. There are no build in 
editors, menus, or graphical aids. The documentation 
consists of three volumes and is sufficient to get staxrted 
on small to medium size systems. Little advice is given 
on building large systems via ROSIE. Finally, no formal 
support is availabla fo. ROSIE. Rand Corporation does 
consult on ROSIE but does not support it in a marketable 
way. 

Additional descriptive Information is available from 
Rand Corporation (14, 15, 16). 



ERIC 



57 

64 



Appendix D 

Expert Systems: 
Implications for the Diagnosis 
and Treatment of Learning Disabilities 



60 

67 



28 



EXPERT SYSTEMS IN CIVIL ENGINEERING 



Sincm tiiLn sys^wa would used for dwnons^ra-tlon 
.p\irpoa«s, 'tbe por-tabllil:y of a PC-based program appeared 
attractive. For this saae reason at demonstration, the 
ability to produce "run-only" versions ot the expert 
system was considered an Important factor. After some 
deliberation, M.l was choosen for system development 
because the potential was there for building an expert 
system with the forementloned characteristics. However, 
one must be aware of the surrounding circumstances. 
First, since we are an education Institution, all of the 
evaluated tools have been aqiilred at a greatly reduced 
cost or no charge at all. Second, Utah state University 
has available considerable In-house expertise In building 
•xp^rt. systems as well as software engineering In general. 

I«a«tJ.y, some of the newer PC-based tools, such as Insight 
II, were not available at the onset of this project. If 
carried out today, the doclslon of which tool to use for 
building this demonstration expert system would probably 
be different. 

Food For Thought: 

It Is of utmost Importance for any civil engineer who 
wishes to build an expert system to realize that one can 
learn to be a Icnowledge engineer rather rapidly. In fact 
many civil engineers are already knowledge engineers 
without even knowing It. Most engineers are gulte good at 
•retracting complex loiowledge and based on scientific 
assumptions, produce simplified heuristics 
(Rules-of -thumb) . Obviously, one can not be just a 
knowledge engineer and expect to become a civil engineer 
overnight. For this reason, civil engineers should 
seriously consider building their own systems with the 
help of flexible and user friendly tools before hiring 
those who are not familiar with civil engineering 
problems . 
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Sinn this lystu vould bi used (or dnonstratlon 
.puiposu, tbi porttblUty of t pc-bued prograi appaand 
attraetlvt. for thli m ruion o( dnonstratlon, tbi 
ability to pioduM 'nn-only* versions o( tbe exptrt 
lystn vas considered tn liportant (actor. After sou 
deliberation, X.l vas eboosen for systea developaent 
because tbe potential vas tbere for building tn expert 
systei vltb tbe foruentloned cbaracterlstlcs. Hovever, 
one wst be avare of tbe surrounding clreuistances. 
First, since ve are an education Institution, all of tbe 
evaluated tools bave been aijiilred at a greatly reduced 
cost or no charge at all. Second, utab state University 
has available considerable In-bouse expertise In building 
expert systau as veil as softvare engineering In general, 
lastly, some of tbe never PC-based tools, sueb as Insljbt 
II, vere not available at tbe onset of tbls project. If 
carried out today, tbe dnclslon of vblch tool to use for 
building tbls deaonstration expert systei vould probably 
be different. 

Fool For Thouihl; 

It Is Of utaost iaportance for any civil engineer vbo 
vlshee to build an expert systea to realize tbat one can 
leani to be a knovledge engineer rather rapidly, In fact 
lany civil engineers are already knovledge engineers 
vltbout even knovln? it. Host engineers are quite good at 
extracting coaplex Medge and based on scientific 
assuaptlons, producii slaplified beurlstlcs 
(Mes-of-tbiaib). Obviously, one can not be just a 
knovledge engineer and expect to becoae a civil engineer 
flvernlgbt. For tbls reason, civil engineers should 
seriously consider building their ovn systeas vlth tbe 
help of flexible and user friendly tools before hiring 
those vbo are not faailiar vltb civil engineering 
probleas. 
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Implications for the Diagnosis 
and Treatment of Learning Disabilities 



EXPERT SYSTEMS: IMPLICATIONS FOR THE 
DIAGNOSIS AND TREATMENT OF 
LEARNING DISABILITIES 



Alan M. Hofmelster and Margaret M. Lubke 



iUtftoct Application of artlflctal InteUigence to the problems of education is a 
relatively recent endeavor. This article wlU focus on one of tiie most promising 
^^^l^^^. mtdligence - eMpert ^e«a technology . and some of tiie 
characteristics tiiat make expert systems -InteUigenf . Selected present and poten- 
tial a^pUcatilons of expert systems to tiie field of learning disabilities are presented 
along with examples of specific expert systems. 



pcrtise, devebping a problem-solving framework, 
and eventually making the knowledge available 
to others through a computer-based expert 
system. The expert system usually gathers infor- 
mation firom the user in a dialogue format that 
simulates a consultation with a human expert. 
Many expert systems are designed to explain their 
line of reasoning in everyday English rather than 
computer code. 
Reasoning Procedures 

The expert system's reasoning proceduies, 
sometimes refened to as the inference engine, acts 
upon the combination of user-supplied informa- 
tion and information contained in the expert 
system^ knowledge base. 

To facilitate the interaction with the inference 
engine, the knowledge base is organiT^d into rules, 
consisting of two components, an If component 
and a "^hen" component. When the conditions in 



AppBcation of computer technok>gy to the field 
ol learning disablfities has taken a variety of forms, 
the most common being computer-assisted in- 
s^ctton (CAI), computer-managed instruction 
(CMI), and computer-assisted testing (Hofmeister, 
1984c). To a large extent these appUcattons repre- 
sent reasonably well-developed procedures that 
existed before mfcrocomputers, but had to wait for 
the wkle^read availability of this technology to 
achieve their present popularity. More recently, a 
new computer technoIo>)y —the expert system^ 
has been developed. 

A fiekl within artificial intelligence, expert 
systems technology is concerned with the use of 
computers to capture and disseminate human ex- 
pertise. Typically, expert systems have proven ef- 
fective in medicine, geok)gy, chemistry, engineer- 
ing, and business. However, educators have le- 
cenljy begun to show an interest in this technolo- 
gy, particulariy as it can be applied to the problems 
associated with learning disabilities. This article 
reports on present and potential applkatlons of 
expert systems technology to diagnosis and treat- 
ment of learning disabilities. 

EXPERT SYSTEMS 
Knowledge Engineering and ''Expert 
Systems" 

Knowledge engineering is the term often used 
to describe the process of capturing human ex- 
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Main rule for timeout 

If A) behavior tr be reduced 
and B) should be aduced quickly 

I n^ ^Jf situation OK for tiineout 

J and D) characteristics of child OK for timeout 
then recommended procedure .s timeout 



Lh oi'ite^^.n^ child in timeoutU^ 

f?^n ih^"!"" ' '^^^ in self-stim. behaV^ 

then characteristics of child are OK for timeout 



or hi rhiiH ."1'" H° !° ^'"1®''"^ assistance 
^nH A ^°^c®^ 90 to timeout 

and c) child will stay in timeout 

thin ^i^^''^^^"? be forced to stay in timeout 
then It IS physically possible for child in timeout 



Question to user: -When the student is placed in a timout situation, 

I? he/she likely to stay until he/she is told to 
return to normal classroom activities?" 




Figure 1. F^ure 1 illustrates that intermediates within if-then rules (A B C D 
wth^n the main rule for timeout) can be made up of oZ rutei which 
Son have intemiedlates. At the bottom of^e pai is a 
flSlna bv t^p cf "^^in this segment of Infomiatlon 

gathenng by the system. Only a single path of rules and Intermediates 
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Formative Evaluation in the Development 
and Validation of Expert Systems in Education 
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lalrodadioa 

EAicaion have beca compmively slow la applying cxpeit 
pril«M tactaology to ia«^^ 

IB OM tcbooto. Thcie have aoine veiy food leasoitt for this 
dalay, not the kast or which if rt^jciik»'s Uffliaed leaetfch and 
devclopmctt budget. irthelindtfrAitaoiiicasoroducaionaitto 
ba tned efrectivcly hi the develop naai jf expett tyttema, their 
efforts inM be ratioaallygiihMandpiogieaelvelyiell^ 
wayiodottitisiotiaeitaeafChaodilevelopaeatiiiodehi.Theae 
moddt ihoeld provide a theocctkal baae. aeive to Cfystatlize 
pe« wcccsiM effom, and caiiiie that fbiure modela and their 
product outcotaes aie buih oa tealed ptacticea. 

Geaeral-purpoae models cxitt 10 guide the development and 
yalidatum of expert systems. One of the bencr-known modeU 
lisis five major stages hi the developmeat or en expert system, 
namely, idcnrification, ctiacyptualiTMion, fonaalizaiion, im- 
plemeaiation, sad lestirig (Hayca-Roth «r «I903). 

Thtm g caer ai p tap o ^ modeb have coasMerable value for 
rddtog initial p laa a i ng . 11m vahieoriheae models diminishes 
ss implementaiion appRMcfaes. Hie need for mote pnibiem- 
spedftc models lepieseatt a natural evohitioo. Hie difTefemia- 
liM in modeto OGcun as we aneinpt to guide the developincnt of 
diffeifni types of expert systems with differeat theoretical 
bates. Coostnint*besed and stnictuied-selectioo approaches ID 
expert systems may call for difline;^ in specific product 
deveiopmem proceduie ^ffefcntiation in devetopmeni and 
validation proctluies may also occur as we cterte expert 
^ritems in diffeient disciplUies. Hiis differentiation should not 
be constnied u « move towards partinnship among tlw 
disciplines. The quality of all expert systems product develop- 
ment efforts can only be enhanced by experimeniatioo with a 
range of product deveiopmem coostmcts and their supporting 
tools. • 

This article presents and discusses formative evaluation, a 
key component of most educational research and deveiopmem 
modeb. As educatofs search for discipline-specific modeb to 
provide more direction to the deveiopmem of expert systems in 
education, formative evaluation must receive more serious 
coosideraiion. While the dau-based outcome or validaiioo 
testing h often approached systematically, the preceding 
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rormauve «valualions oUcn receive leu syslemalic aiicntion. 

Formative aud tumnuiClve stages: ae overview 
The ro/t'J offormativt ami nimnruivt evaluation 

In a landmark article on prognun cvalualitm pnclices in 
eUucaiion, Scriv^n (1967) uieili the terms **romiative** and 
"summative** to difTeienliate two ttifes in the dcvclopnteni and 
evaliMtiofi of an edueatkmal progran^ yTpoxluct. In romuiive 9 r 
evaluation, the primary questian cs '*What do I have lo do to 
inipnive the productr la aummative evalualtim. tic primary 
question is **Huw well does tha pfuduct wotkT 

Wbeic a developer stresses the difTeicnoes between sumnui- 
tive and fonnative cvaluatioa pfBCttoes, concern for formaiive 
procedures iooeases. Histofk«lly. the importance of such 
formative procedures had been larfdy ignored except for a few 
ohiervatioos, such as die foUowiug by Cronbach (11963): 

Evahiadoo mad lo improva itm count whiki U h Mill fluid 
comrttum man to dw impmw ina nf education diao evaluaUoa 
UMd 10 ippraiie a product already on iteauMlwt. lp.67S.i 
The coooepls of formatJ'/e and suromaeive evaluation are now 

importam compooenu of most modeb used Jii educc>kmal 

product development and validadoo (Bofg aW -Gat: 1979; 

MariUe 1967; Hood 1973; Hcrfmeister 1975; Sanders tnd*«) 

Cunningham 1974). 

la chafonnitivost^, ia v estmBi i u hi*>rfishtog^ the product 
are triiaimal.l^lestiBgUrestrkted CO the intense obaervatioa 
of smaU samples of diB taifei popidation 19 deienmne reasons 
for product malftmctkm, CoMukaat critiques and ^faer similar 
cvahiatiottpfacticcsoccurasaMly as possible, before diere is a 
m^ flscal and ego iavesmu&t bi die pielimbiry produ;;t 
chsracteristica and ooolem. 

Hie summadve stage follows dM formadve stags. Only when 
die formadve stage U largely compleied are rm^or investroenu 
made ia die appearance and ''packaging-' of die product. 
Summative field testing usually iochides much larjcr simples of 
die ttrget pop ulation, and die associated evaluation procedures 
are concenied with asseiimrnt ofd»inalotacomcsntt>>er than 
ttudyiag product functioning duough die more intensive and 
intrusive monitoring practices orformiliveevaluatk».Oeneral- 
iiation and experimental cootiol are fsr non importam con- 
ccms bi sommalive evahmdoos dwi bi formadve evaluations. 

To discrimfaiaie between formadvo and summative evalua- 
tioo acdvides. die mies of die evahiation biformadon, not die 
spedAe dala-gsdiering tools, are bnpoftaai. The respective 
roles have been summarised by Borich (1^74) as follows: 

lafonnatiaa for program itvision is perhaps the sin|le miMt 
impartaoi ch a iacteri sd c oTfomativi evakwioa. while iafoniMikNi 
for pragram adopdoo is dM siagia BMMI inportaot chaiacicrirJc of 
summidvc evahndoa. |p. 2721 

The fermallvt Stage 

Major compomms of the/omaiive siagg 

Two of die major acdvides of die formative stage are die 
design of die system and die devclopmem and r^isioo of a 
series of prototypes, la designing die system, constdoaUon is 
given 10 such questions as 

(a) What type ofproblems should die expert system addreu7 

(b) What type of faiformadoo output should die expert 
systema provide? 

(c) Who will use die system? 

(d) Under what field conditions will die system be used? 

(e) What software and hardware tools hold die most 
promise? 

In die development and revision of a series of prototypes, die 
initial emphasia is 00 individual compooenu or modules. These 
individual modules must be progressively revised to ensure that 
performance is consistent widi specifications established in die 
design stage. If opportunities arise diat allow die developers to 
improve upon design specifications, such opportunities should 
be exercised. Such is die nature of formative processes. In the 
latter stages of prototype developmem and revision activities, 
die emphasis shifts to die relationship among die different 
modules, and dien to die performance of die total system. 

Design spec^kationM and knowletise enghieering 

The relationship between design specification activities and 
prototype developmem and revision is con^lex and interactive. 
There U certainly not a simple linear lock step rehuionship. It is 
tempting, frum a management viewpoim, lo complete the 
design speciftcaliotts and dien not look back as the prototypes 
are devek)ped. Such diinking ignores di^ very nature of 
knowkdge engineering. Design specificatiorftnust be based on 
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our koowMse of the problem at a point in time. Often, the very 
proeeu of developing the knowledge base incteaaes our 
w«to«Mdlng of the problem. Thla inereased umtotanding 
coyld lead to an enhancement of the design specificaiiofu ofihe 
product. TWs interaction between design and devctopment hu 
been advocated by Nocentini ( I9SS) who noted, -| koow of no 
case where a »iAwait project has been succeufully completed 
by following the original plans laid for Ap. 1 22)«** 

Prtdtwiopmenmi actlvitUt 

The previously mentionea fomuuive activities uit all prcni- 
ucd on the aasuroptiuo thai an cipcn syKtvni is on apnitinriiiiv 
ffSS?* ^ problem. Sanden and Cufminghani 

*^ -prodevelopmental- activities should pre- 
cede the initial product design activities. These predev«lopinen- 
talacuvitesbo^ 

dtfough togicaf and empMcal analyses. For example, if the 
problemia a lack of expert assistance to hiterpret complex 
ro|Ulai|oiis, it might be moro appropriate to simplify the 
regulation so dsai expert asaistanceia not needed/^ 
Fcrmarivegvahiathmproeediiru 
F^wmaihfecvaluitionptMicesstitsadiecoUectioiiorcvshia- 

dwtoptag, ami modifying pmotypea. The followiiVaro 
examples of aone of the tvahmtion practices 
10 dnve the product development processes. 

Conwteir ffvinv €f tool sHtctkm ami gfmrai tyjitm 
Mracfcrisfirj. This is done to ensure that the systcma 
appropriate and that the major systems characteristics are 
cottsistcm with the goato. 

Cofuntotf iNElidaHm qfshe primary ruits tftite ktmrinint 

Am. As the knowledge beae b developed, a range oTevaluatkm 

promircs b availabte 10 ensure thai die knw 

andifjwtamfortheproWeinsaddressedbythesysie^ 
knowtedgecm^ 

wrttog far stwmive evidence In synih^ 

HMteic kMwIedge can be ooss^hecfced by seeking c«^ 

Arming opinions from additional experts. 

AsarxsmMT €f Mtltcud rmotiiai pronsm. As tfie knowl- 

eje base and Msocialed reeaoning prf«edurea are dev^^ 

die acoiracy of selected system decisk)iMnaking proc^^ 
««bed|ecj^^ 

problems. Each of the leal pioblema consists of wcr input 
information and die associated oMoo^ 

if die rules end die associated leaaoning piocessea ait ncouate 
Ttaetestwaniplttcanbedevetopedjofc^ 

expanded during die formative stage. 

These test problems and dieir associated ouicosnes also 
provide a vehicle for obtafati^ input from additkmU experu. 
fhr user hyut taf onnation of lest problems U givno (o these 
expats. TTiey are asked to provMe outcome information for 
Mch of die lest problems. By comparii^diis outcome informa- 
two wuh disc provUed by die devekipmem team, ndditiooai 
information on die validity of die system can be obtiinfid. 

Prriimiaary osteMMmmt ufiatr rtiiaitUHy. Using n smsU 
bank of actual IkM problems, preliminary bifarmalion on oser 
reliabdny can be oTitaioed. Thla bank of problems slioutd be 

represeoiaiive of die actual target problems die system 
designed to solve. 

Two aspects of user reUabiUty sbouM be evaluaied: <a) 
Mmitteicy of outcome by die same user over time ami (b) 
consisiency of outcome when diffeieni users attempt to aolve 
the saflM problem. These assemMm procedures should provide 
mformation on such fneton u ambiguities in coosuitation 
q^Mim by d« system and problene in extracting lnr«^ 
rrom the uaar. The educational leseaich procedufts saiocisted 
widi die coefOcUni of stabUity and inierobserver i&reemem 
measuics (Bofg-andOaU 1979) shouM provkle a nngcof dau 
collection and data analysis procedures for cooductifli evalua- 
ison of user reliability iioues. 

P/ogrentfvc nfinemtm luiag /MJ prMgms. Once tiie 
nliabdity of data entiy and odier user proceduira Ijs been 
established, die syrem U progreuively reAned. Field noblems 
are syttenMticaUy sampled in smaU groupa and Use expen 
system eevisdd. based on die system's response to each group of 
proNeias. To ensuru a ryiiematic approach to die saitAling, die 
target population of Add problenv is subdivided tolo UifTcrcnt 
problem types andasmaUgroupcfSeld problems sartipled from 
eachtyps. H&eresuhsof'tie consultation for each ssmple group 
are exair.^^d by knowledge engineen and knowledge base 
consulta:u9 so ensure dial die outcome of die consuliation is 
appropriate and die ioieraction between system nnd user / < 
eincient. • *^ 
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«« P«oWemi 10 help emuie thai 

««wy allMihe dcctsioa.iniUn| pfoeenet ia other utu. 

P'y*".y* jf ewwuiiteitd thai b not adequately 
«PJM«ted in the bank or te« ptoblemt. the baiTor leM 
probkim b rappleneMed or reviaed to 
•»Wjt« aU the lnipait« deeiaioo-maki^ 
^SL^Saaf V«K*u h GOMimied until all 

.'^!S^r^..^,'^ Populatfan of pfoblema have been 
»«npWi^dteiinplemeniatkHiofnewfleldprobte^ 

ftyehohilcalcUmau 

When cojductini roni»Jtlve evahiatkm aciivitiei, the re- 
eearcher'a diae^ ot^ectivtay i. con««>dy d»lta«^. to 

pncedum and ntoic on cnaCvky and ttm level of obieeiivliv 

prartpUve pjDceduwa aaaodaied with lioiiMUve evaluation, 
tamttww to iMowledie eii^iyeri^ 

I^!5^S!S£iir!!!2!^***°* -Id the beat loluitoni. die 
•esuli or ndndleai adheicnce 10 a ■tynena" ndpe. 



wnnadve iiaie are (a) a predevelopinew fBvlew of the need for 

r!!3?^**'"**»'"«^«««taof proioiypea. Inthjirniib 

onen n^lect theae foimaiive pncciaea or emneoualy atiuaie 

wiU ilao be app«p,la» tor a«a*«tai Sr^^ 

cooimltted when die (»iniB«lwovahiailoBeiMih»i«aco^ 
bonaamoni pMially developed ippwad^ artpiwZre 

PotentJaUy poweifld proeedMre to obacuiity. Too often dw 
coiwuinerofd>ere»ean*rocuie«onlhedifretwice»ino«comei 

leBwfaaioa. 3id ad. I wn w an. Inc. New Yoric. NY. pp. 7s. 
BoeiCM. 0. D. (£«tor). 1974. BvahMdatcdycatkaJ^ «d 

IVdbWH OoMpaKy, Inc. Rcadiat. MA, pp. 3-29. 

Utaw ictMcy panicipatiaa and hiwJIcapped popuMoat: An 
•iKMMaicrmcaidi aeadi. £UMty P. J. Vcri^udj E 

Hooo^ D. 1973. EvahiaUen aapn Ibrmaiar bbwalary pndueu. I» 

*T J. K. HaavhiU. Uaivmiiy ofOicioa, Eutna, OR. 
MAwaa. S. M. 1967 Empirical ie«h57p™,Si. /, Pwr"-«td 
nmucuoa. Sbty-iblh yeartnok of die Niliaaal SoriHyfoAe 

NocfHTiM. S. IMS. Hie pbaniag riuiU Dalamalion. 31(1), pp. 

l974.PomiaUva«vahialioa: 
SelecUat ladadquei a«i pnceduiet. /« Bvabadai cduMtad 
Jf^nim aad pndyda. BUitd b, O. D. Borieh; Uuadoad 
T«da«*w MUicatkiM. Eatkwood aiffi. NJ. pp. JTOlIsuT 
Scaivai.. M. 1967. The nMlMidolaiy of evaluallooVPtt«ecUm of 
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Accuracy of CLASS. LD2: 
An Expert System for Classifying Learning 
Disabled Students 
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Accuracy o£ Cla8S.LD2: An Expert System for 
Classifying Learning Disabled Students 
Elizabeth S. Hartindale^ Joseph H. Ferrara, and 
Brian W. Campbell 
Utah State University 
Department of Special Education 



Running head: ACCURACY OF CLASS. LD2 



ERIC 



Accepted for publication in Computers in Hum&n Behavior (in press) 



95 

104 



Accuracy of Class. LD2 

2 



Abstract 

This study detarmined the accuracy of Class.LD2r an 
•Xpert systen toe classifying learning disabled 
students, of 264 student files, 78 files were chosen 
based on disagreement between oul tidisciplinary team 
and Class.LD2 decisions regarding eligibility for LD 
placement. These 78 cases were evaluated by three 
individuals expert in LD classification, who made an 
eligibility decision for each case. Their decisions 
were compared to those of the expert 8ystem2« 
Analysis of the results indicated that 1) Class.LD2 
was in agreement with the experts more often than were 
the placement teams, 2) the expert system*s decisions 
were significantly correlated with those of the 
experts, and 3} in those cases in which the experts 
were in unanimous disagreement with Class. LD2, it was 
shown that Claas.ti02 conformed strictly to state rules 
and regulations in making its eligibility decisions. 
The use of Class.L02 by placement teams in order to 
encourage more data based decisions and to limit 
overclassification errors is also discussed. 
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3 

Accuracy of Claaa.LD2: an Bxpcrt System for 
Claaaifying Learning Disabled Stndants 

Special education placement teams must make 
decisions regarding the identification of handicapped 
children and their eligibility for special education 
services. Current data suggests the judgments made by 
placement teams may not be accurate. Ysseldyke (1983) 
contends that half of the number of identifications in 
the area of learning disabilities may be inaccurate. 
Hofmeister (1983) reported an 84% increase in the 
number of pupils identified as learning disabled during 
the past few years. Placement team inaccuracy may be 
related to this dramatic increase in the number of 
pupils identified as learning disabled (Algozzine s 
Korinek# 1985; Algozzine/ Ysseldyke s Christenson/ 
1983). 

If placement teams are doing a poor job of 
applying eligibility criteria, handicapped and 
non handicapped children are not being well served. 
First/ nonhandicapped children may be receiving 
inappropriate services. Second » if nonhandicapped 
students are qualifying for special education/ there 
is less money available to meet the needs of the 
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handicapped, and essential services for children with 
other handicaps are being reduced to pay foi the new 
learning disabled students (Sabatino, 1981). 

In some states the problem of overclassif ication ' 
has reached crisis proportions, state education 
agencies have responded to this problem by placing 
limits on the number of studercs who can qualify and 
receive funding for services in the learning disabled 
category (Boyan, 1985; Reynolds, 1983). Such limits 
may solve the immediate fiscal crisis. Unfortunately, 
they subvert the intent of P.L. 94-142 in that they 
discriminate against handicapped children who happen 
to live in schools with an unusually high handicapped 
population. At the same time, they encourage 
aggressive and "imaginative" administrators in 
districts with a low percentage of handicapped 
students to find pseudo-learning disabled students to 
fill their "quota" and obtain additional funding. 

An alternative to state Imposed limits is to 
improve the accuracy of the decisions made by 
placement teams. Research on the current functioning 
of placement teams indicates that their inaccuracy may 
be the result of a flawed approach to placement 
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decision making. P«w school placttment teams employ a 
systematic approach to determining eligibility 
(Ysseldyke, Algozzine & Mitchell, 1982). As a result, 
a great deal of data describing student performance is 
collected, but much of it is technically irrelevant 
(Thurlow & Ysseldyke, 1979). Teams spend about 30% of 
their time discussing these questionable data 
(Ysseldyke, Algozzine, Rostollan, & Shinn, 1981). 
Furthermore, individuals discussing these data are 
likely to use language which is unfamiliar to 
aoneducators (Ysseldyke, 1983). Considering this 
unfortunate state of affairs, it is not surprising 
that Ysseldyke, Algozzine, Richey, and Graden (1982) 
reported that there was little relationship between 
the psychometric data presented to placement teams and 
the eligibility decisions made by those teams. 

Two changes in the decision making process might 
reduce the frequency of errors. First, teams should 
follow a rational series of steps which encourage the 
examination of relevant data. Second, teams should 
use a set of regulation-based rules to make judgments 
about placement data. 
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On« way to bring about these changes involves the 
application of expert system technology. An expert 
system is a computer program which attempts to 
replicate the decision making and problem solving 
skills of knowledgeable and effective human experts 
(Hofmeister & Ferrara# in press). If an expert system 
which behaved like a knowledgeable # systematic expert 
in the area of eligibility decision making could be 
developed/ it might serve as a useful tool for 
placement teams. An expert system might force a 
rational step-by-step approach which would result in 
judgments which were directly related to data and 
regulations (Hofmeister & Lubke# in press). 

Before such a system can be used by placement 
teams its judgement must be shown to be competent. An 
expert system^ Class. LD2 (Ferrara & Hofmeister/ 1984) # 
has been developed at Utah State University. The 
purpose of Class.LD2 is to evaluate student 
eligibility for special education placement (Ferrara# 
Parry « Lubke/ 1985). The degree to which Class. LD2 
outperforms placement ceams should be related to its 
potential value as a tool for increasing the accuracy 
of placement team decisions. One way to evaluate 
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Class.LD2*s performance is to compare its judgments to 
those of placement teams as veil as to the judgments 
of acknowledged experts in the area of LD 
classification and placement. The purpose of this 
study was to make these comparisons. 

This study addressed the following research 
questions: 

1. Here the decisions made by Class. LD2 and the 
placement teams different? 

2. ffhen placement teams and Class. LD2 did not 
agree# what was the nature of the disagreement? 

3. In cases where placement teams and Class. LD2 
disagreed^ whose decisions did experts judge to be 
correct? 

4. What were the characteristics of cases where 
experts agreed with the placement team instead of 
Class. LD2? 

METHOD 

Class. LD2 

Cla8a.LD2 (Ferrara & Hofmeister# 1984} is an 
expert system designed to provide a second opinion 
regarding the classification of students as learning 
disabled (Ferrara et al.# 1985; Hofmeister & 
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Lubk«, in press). class.LD2 uses a dialogue or 
consultation format to obtain information from Cha 
user. That information is compared against the rules 
in a preprogrammed knowledge base and is used to make 
a classification decision. 

CLASS. LD2's knowledge base contains approxin^cely 
600 "if-then" rules. The rules are based on P.L. 94- 
142, Utah state regulations, and current literatiire 
describing best practices in the area of student, 
assessment anv- placement. As questions are generated 
by the program and answered by the user, decisiona are 
made based on a comparison of the answers to the 
information in the knowledge base. 
Sample cases 

Data from 264 files which described potential, 
special education students were used to evaluate che 
performance of Class.LD2. These files were selected 
from three school districts. District 1, located in 
Idaho, provided 150 files. District 2, located in 
Utah, contributes 54 files, and District 3, located 
in Wyoming, contributed 60 files, of the total ntinber 
of files, 110 of the students were judged by placement 
teams to be eligible for placement as learning 



102 

111 



Accuracy of Class .Loa 

9 

disabled. Th% ramaining 154 cases vera either judged 
to be nonhandicapped or eligible for services in 
another disability area. 
Data Input 

The data from each student file was provided to 
Class.LD2 in a consultation format. Special education 
teachers familiar with each case entered the data from 
Districts 2 and 3. District 1 data were entered by 
graduate students pursuing a PhD in psychology. The 
graduate students worked as consultants to District 1 
and were familiar with each case. Neither the 
graduate students nor the teachers were involved in 
the development of Class. LD2. They were provided 
instruction on Class. LD2 prior to their use of the 
system, fiecords of each data entry session were kept. 
These records contained the questions asked by 
Class.LD2 and each response made by the users. Data 
entry sessions took about 8 minutes per case. 
Disagreeing Cases 

Nhen Class. LD2 had evaluated each student's data, 
its judgment was compared with the judgment made by 
the placement team. Cases where disagreement occurred 
were identified. 
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Claaa.LD2 doaa not normally provide a dichotomoua 
judgaant (i.a./ LO or not LD). Rather^ claaa.LD2 
will provide aa nany as eight poaitive hypotheaea 
regarding the atudent'a area of diaability. Certainty 
factora of -100 through +100 are then aaaigned to 
theae hypotheaea. in order to judge each caae aa an 
agreement or diaagreement, Claaa.LD2'a judgmenta had 
to be viewed aa dichotomoua rather than continuoua 
data. If claaa.LD2 aaaigned a certainty factor of 
•t-50 or greater to the LD hypotheaia, the atudent waa 
viewed aa being judged by the ayatem to be eligible 
for LD aervicea. Therefore^ if a certainty level of 
•1-49 or lower waa aaaigned, the atudent waa conaidered 
to be not LD. 
Expert Evaluator 

Caaea where Claaa.LD2 and the placement team 
diaagreed were aubmitted to three experta. Expert 2 
ia a achool paychologiat in a large Utah achool 
diatrict. Expert 1 and Expert 3 are employed by the 
Utah state Office of Education. A principle component 
of all three experta' work involvea aaaiating local 
achool diatricta in making placement deciaiona. 
The experta were given data aheeta which 
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connain^d th^ same information that was provided to 
Class*LD2. Based on this informationr the experts 
were asked to judge stud^ ts as either eligible or 
ineligible for LD placement. 

In cases where the experts unanimously disagreed 
with Clasa.L02's judgements the intermediate 
conclusions of Class.LD2 were examined. Specif ically, 
three critical attributes were examined: (a) IQ 
level, (b) a discrepancy score based on the student's 
actual grade level and, (c) a discrepancy score based 
on an estimate of expected performance. 

Results 
Placement Team Decisions 

0£ the 264 files checked with Class.LD2, there 
were 78 cases where placement team decisions did not 
match Class.LD2*8 decisions. In other words, Class.LD2 
and the placement teams were in agreement 67% of the 
time. Table 1 shows a 2x2 table illustrating the 
relationship between Class.LD2*s decisions and the 
placement teams* decisions. A phi coefficient of phi«.46 
was calculated using these data. 



Insert Table 1. about here 
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Tables 2t 3, and 4 show the ralationship batwean 
tha placanant decisions of Districts 1^ 2, and 3 and 
those of Class.LD2. A phi coefficient of phi«.27 was 
calculated for District 1. A phi coefficient of 
phi«.56 was obtained for District 2. Finally, the 
relationship between Class.LD2's judgments and those 
of District 3's placement team is described by a 
phi-. 24. 



Insert 


Table 


2. 


about 


here 




Insert 


Table 


3. 


about 


here 




Insert 
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4. 


about 


here 



Nature ot Disagreements 

There %iere 78 cases where ClasSeLD2 and the 
placement team did not agree. These 78 cases can be 
divided into two categories: (a) Type A cases# where 
the placement team said the student was learning 
disabled and Class. LD2 said the student was not, and 
(b) Type B cases, where the placement team said the 
student was not learning disabled and Cla3s.L02 said 
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the student was learning disabled. There were 68 Type 
A cases and 10 Type B Cases, when disagreement 
occurred, then, 87% of the disagreements were a result 
of either team overclassif ication or machine 
underclassification. 
Accuracy of Decisions 

The 78 cases where disagreement occurred were 
submitted to three experts. When taken together, the 
experts made 234 (3X78) judgments. Figure 1 shows the 
experts* total agreement and disagreement with 
Class. LD2* 



Insert Figure 1. about here 



Table 5 shows the relationship between the 
experts* placement decisions and those of Class. LD2. 
Using the data shown in Table 5, a phi coefficient of 
phi».40 was calculated. This phi is associated with a 
chi-square of 36.62, which is significant at the 
level required to reject the null hypothesis that 
there is not a relationship between the experts' 
judgments and Class. L02's at the £<.001 level of 
confidence. 
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Insert Table 5. about here 



Table 6 shows the relationship between 
Class.LD2's decisions and all cases where the three 
experts agreed. There were 58 such cases. Phi».63 
and a chi-square of 23.38 were calculated to describe 
this relationship*, Once again we can reject the null 
hypothesis that there was no relationship between 
Class.L02*s judgments and the experts* judgments at 
the £<.001 level of confidence. 



Insert Table 6« about here 



Table 7 shows a correlation matrix describing the 
relationship between all five judgments of student 
eligibility (Class. LD2# placement teams, and the 
experts). The decisions of Class. LD2 and of all three 
experts are inversely related to placement team 
decisions, it should be noted that only cases where 
placement teams and Class.LD2 disagreed were used to 
calculate these correlations. 

Insert Table 7. about here 



EKLC 



V7 

108 



Accuracy of Claaa«LD2 

15 



Characf riatica ot Caaea 

Tha human axparta' judgmanta vera unaninoua in 
only 6 of tha 78 caaaa of Claaa«LD2/placafflant taain 
diaagraamanta. Tabla 8 ahova Claaa«LD2*a intarmadia ta 
concluaiona for thaaa caaaa on thair critical 
variablaa: IQ, aatimatad diacrapancy and actual 
diacrapancy. In all aix of tha caaaa vhara Claaa«LD2 
and tha axparta diaagraad, Claaa*LD2 mada an 
undarclaaaif ication arror* 

Inaart Tabla 8. about hara 

In aach caaa, tha atudant*a data failad to 
aatiafy ona of tha LD cutoff lavala aatabliahad in 
Claaa«LD2*a knovladga baaa* Thaaa cutoff lavala 
includa a mininum IQ of 83 or 84 (dapanding on tha 
atandard deviation of tha IQ taat uaad) and a 40% 
diacrapancy batuaan axpactad and obaarvad acadamic 
parformanca. 

Diacuaaion 

Ovarall Aqraamant 

Tha raaulta c .. la atudy auggaat that Claaa*LD2 
and placanrnt taana tandad to agraa on moat (74%) 
caaaa* Thara vaa^ hovavar^ anough diaagraamant to 
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justify an avaluation of the 78 disagreeing cases* 
Evaluation of Disagreeing Cases 

The 78 cases in vhich Class«LD2 and the placement 
teams disagreed can be viewed as difficult cases* in 
these situanions Class.LD2 clearly and consistently 
outperformed the placement teams. The few cases in 
vhich the experts all disagreed with Cla8s«LD2 were 
marginal. Furthermore, Class.LD2*s decisions were 
defensible and its reasons for making those decisions 
vere clear. 

Cautions for Interpreting the Data 

This study uses a portion of the data vhich were 
collected during the formative evaluation of 
Class«LD2* As a result, both the data and the 
analysis are gaared toward product development rather 
than a summative report on the performance of the 
product. The reader should be alerted to number of 
conditions which limit the degree to which the 
conclusions of this study can be assumed to generalize 
to a larger population. 

atah Guidelines for Idaho and Wyoming Students 

The reader should recognize that Utah's LD 
classification criteria were used to evaluate files 
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froB two oth«r states. Class.LD2 was mov Hk«ly to 
agree with placement teams from District 2 (the Utah 
District) than with teams from the Idaho and Wyoming 
school districts. The placement teams' degree of 
inaccuracy which might be inferred from these data is 
probably inflated. 
Fence Sitting was not Allowed 

In this analysis, Class.LD2's advice was 
transformed from continuous data to dichotomous data. 
As a result, some error was introduced. For example, 
Class.LD2's consideration of one student's data 
resulted in an advice statement which suggested that 
the student could te classified LD at a -1-48 level of 
confidence. Using a +50 cutoff, Class.LD2's advice in 
that case was considered to be not LD. In the field, 
a +48 certainty level would probably not be 
interpreted as an absolutely negative response. Most 
professionals would recognize that Class.LD2 was 
suggesting that, in this case, an LD placement would 
be marginally defensible. The continuous nature of 
Class.LD2's advice would make its judgments useful 
even in those cases where it disagreed with one or 
more of the experts. But in this study near misses 
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did not count. 

Controvrsial issue s; Class. LD2 is Smarter than it 
Looks 

Several elements of the currently accepted 
criteria for LD placement are open to professional 
debate. For example, if a student's learning problems 
can be primarily attributed to environmental, economic, 
or cultural deprivation, that student cannot be 
classified as learning disabled. 

Mot everyone agrees (Sabatino, 1933) that students 
with learning problems attributable to environmental or 
economic factors should be excluded from LD placement. 
Conversely, concern has also been expressed that 
environmentally and economically disadvantaged students 
have been placed in special education programs and that 
this practice could place learning disability 
programs in danger of becoming a dumping ground for all 
e<2ucational problems (Kirk ( Kirk, 1983). 

The experts evaluating the placement 
team/cla88.LD2 disagreement cases were not in 
agreement on the environmental or economic issue, in 
nine cases where there was clear environmental and/or 
economic deprivation, the experts disagreed. Two of 
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thm experts chose to call all nine of these children 
learning disabled. The other expert chose not to 
classify any of them as learning disabled. 

In all of these casesr Class.LD2*s advice took a 
■iddle ground. The program alerted its users to the 
controversial issue and then pointed out that a strict 
interpretation of current state and federal guidelines 
would not allow an LD placement. For the purpose of 
analysis^ Class.LD<:*s advice in all such cases was 
judged as not LD. This interpretation accounted for 
a large portion of the disagreements between Claas.LD2 
and the experts. Clear ly^ the analysis conducted in 
this study was not sensitive to Class.LD2*8 
appropriately moderate advice in controversial areas. 
Dse of An Expert System to Limit Overclassification 

If Class.LD2 had been used by placement teams to 
generate a second opinion regarding LD placements the 
number of overclassification errors might have been 
reduced. The decisions made by placement teams might 
then have reflected a more accurate application of 
state and federal rules and regulations. 

If the use of Class.LD2 does indeed reduce 
placement team errors the need for state imposed LD 
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Halts will bm r^duc^d. Current r^s^arch Is attempting 
to a^asurs thm offset of Class.L02 on plac^n^nt tmam 
porf oraancs • 
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Table 1 

Cia5s.L02 and overall Placement Team Decisions 
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PLACEK£^^• TEAMS 
LD NOT LD 



M 
□ 

CO 



LD 



NOTLD 



64 


10 


68 


122 



N-264 

Total Agreement ■ 54 + 122 - 185 
Total Disagreement -10 + 68-78 
Phi Coefficient - .46 
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Table 2 

Class.L02 and District 1 Placement Team Decisions 



PLACEMENT TEAMS 
LD NOT LD 



CM 
□ 

5 



LD 



NOTLD 



32 


6 


31 


81 



n - 150 

Total Agreement -32 + 81-113 
Total Disagreement -6 + 31 - 37 
Phi Coefficient - .27 
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Tables 

Ciass.L02 and Oistricx 2 Piacemenx Team Decisions 



PLACEMET^IT TEAMS 
LD NOT LD 



M 
□ 

1 

u 



LD 



NOT LD 



17 


0 


16 


22 



N-54 

Total Agreement -17 + 22-39 
Total Disagreement -0 + 15-15 
Phi Coefficient - .56 
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Table 4 

Class.L02 and District 3 Placement Team Decisions 

PLACEMENT TEAMS 
LD NOT LD 



M 

a 

in 



NOT LD 



15 


1 

i 

4 


22 


19 



N « 60 

Total Agreement -15 + 19-34 
Total Disagreement -4+22-26 
Phi Coefficient - .24 
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Tables 

Ciass.LD2 and overall 6(pert Oecisions in cases of 
Team/Class.LD2 Disagreement 



EXPERTS 
LD NOT LD 



M 
□ 



NOT LD 



22 


8 


42 


162 



N « 78 Cases X 3 Experts - 234 Judgments 
Total Agreement - 22 + 162 - 184 
Total Disagreement -8 + 42-50 
Phi Coefficient - .45 
Chi Square - 36.36 
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Table 6 

Ciass.L02 and Unanimous exper? Decisions in cases of 
Team/Cla$s.LC2 Disagreement 



EXPERTS 
LD NOT LD 



I 

O 



LD 



NOTLD 



5 


0 


6 


47 



n-58 

Total Agreement - 5 + 47 52 
Total Disagreement 0 + 6« 6 
Phi Coefficient - .63 
Chi Square - 23.38 
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Table ? 

Correlation matrix: Phi coefficients describing the reiatiomhiK bgtwggn 
the judgements of tganns. Ciass.LCiz. and experts in 
cases where there is tcam/Class.LD2 disagreement 



CL 


E 1 


£2 


£3 


•PT -1 


-.39 


-.59 


-.24 


— CL 


.39 


.59 


.24 






.50 


•79 


£2 






.45 


E3 








•Placement Teams 








"•Class. LD2 








•••Expert 









N - 78 
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* T4ble 8 

Intermediate Values of Critical Variaoies in Cases of Unanimous 



Exptrt/Clas$.LD2 Disagreement 


Variable 


IQ 


Est. . 
Discrepancy 


Actual 
Discrepancy 


Cutoff Point 


65/84 


40% 


40% 


Case # 1 


1QQ 


49.3% 


•39.6% 


Case « 2 


1QQ 


-39.5% 


48.9% 


Case#3 


-84 


54.0% 


55.8% 


Case#4 


-83 


50.0% 


54.8% 


Case#5 


104 


40.0% 


-38.5% 


Case# 6 


95 


40.5% 


•38.1% 



- The critical score which failed to make Clas$.LD2'$ 
cutoff level. 
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Figure Caption 

Flflurej^. Number of agreements and disagreements of experts 
with Class.L02. N « 78 X 3 « 234 judgments. 
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Appendix I 

LD. Trainer: 
Modification of an Expert System 
for Complex Conceptual Training 
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Expert Syatem for Complex Conceptual Training 
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LD.Trainer: Modification of a 
Expert System for Complex Conceptual Training 

Suppose your two-year-old son. Tommy, has suddenly become 
infatuated with the word, "blue." Every time you point to 
something and ask, "What color is this?" Tommy says, "blue." 
You're tired of him calling everything blue so you decide to 
teach him. that some things are blue and other things are not 
blue. Intuitively, you point to Tommy's blue shirt and say, 
"Tommy's shirt is blue." Next, pointing to Daddy's shirt you 
say, "Daddy's shirt is not blue, it's green." You are beginning 
the process of teaching Tommy the basic concept, "blue." 

Basic Concepts 

Basic concepts, like blue, comprise a good portion of any 
language. They are ideas or discriminations that cannot be 
easily described using words. Thus, one must employ examples and 
nonexamples to teach them (Englemann & Carnine, 1982). To 
illustrate, consider that Tommy's parent, as described above, 
used his owi shirt as an example of something that was not blue, 
a nonexample. By continually presenting Tommy with examples and 
nonexamples of blue Tommy will eventually learn to discriminate 
between blue and other colors. When he has this skill, it can be 
said that Tommy has learned the concept blue. 

In recent years, educators and computer scientists have 
joined in efforts to develop computer programs that can assist in 
teaching basic concepts. Most of these programs generate 
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examples and nonexanples of the basic concept they are designed 
to teach. 
Complex Concepts 

Basic concepts may vary along ouly one dimension. For 
example^ color — something is or is not blue; temperature 
something is or is not cold; or shape — something is or is not 
round, in contrastr complex concepts are a) mul tifaceted^ they 
vary along two or more dimensions^ and b) dynamic ^ the way they 
vary along each dimension is defined by a number of variables. 
Furthermore, several of the concept's dimensions may interact. 
Thus, a change in one dimension could impact our judgment 
regarding another dimension. 

The concept, "learning disabled student,'' provides an 
example of a complex concept. Before students can be classified 
as learning disabled, they must meet a variety of criteria. In 
Utah these criteria define three discrete dimensions: 

1) A discrepancy between students* expected academic 
performances and their actual academic performances 
must exist. i 

2) The students' learning problems must not be the result 
of some other handicapping condition (i.e. mental 
retardation, behavior disorder). 

3) The students' learning prr^^lems must not be the results 
of cultural, economic, or environmental factors. 

Thus, as is illustrated in Figure 1, the complex concept, 
"learning disabled student," is multi faceted. 
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Insect Figure 1 about here 

The complex concept, "learning disabled student" is also 
dynanic. To illustrate, consider that in the State of Utah, the 
discrepancy between expected and actual academic performance is 
determined by administering an intelligence test and one or more 
achievement tests. A discrepancy formula is then used to 
calculate a score which describes the degree to which the 
students* academic performances falls below their expected 
performances. if students* discrepancy scores are greater than 
40, they are eligible for a learning disabil i ties( LD) 
classification. This criterion defines one of the dimensions of 
the complex concept "learning disabled student." But suppose a 
student who comes from a home where only Spanish is spoken has a 
discrepancy score of 45. If tests standardized only on Anglo 
students were administered to this student, our confidence in the 
discrepancy score is altered. We are less than 100 percent 
confident that the discrepancy is truly 45 because scores on 
tests not standardized on Spanish- speaking students were used 
the calculation^ The actual degree to which this fact decrease., 
our confidence is unknown and requires a best es imate or 
professional judgment. That is, i ^^gree of unceit:ainty has been 
introduced. Additionally, the in ^tion that th^ stuc^nt comes 
from a Spanish-speak home raises 4 other issue. The learning 
problems of Spanish-speaking students may be du«r to cultural 
factors. The example above illu/$^.cates the dynamic iiature of 
complex concepts. The fact the' the student comei> from a 
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Spanish-sp^aking home alters the confidence ve can have in two of 
the three dimensions that define "learning disabled students." 
This is illustrated in Figure 2. 

Insert Figure 2 about here 

When the best professionals identify learning disabled 
students/ they consider the student's characteristics in all 
three dimensions that define this complex concept. They 
understand that the confidence with which one can conclude a 
student is learning disabled varies with specific circumstances. 
The student's age/ IQ, and cultural background/ as well as the 
quality of information provided by tests and other sources/ are a 
few of the specific circumstances which modify judgments 
regarding learning disabled students. Skilled professionals 
understand that the criteria for a learning disabilities 
classification is dynamic rather than absolute. 

Since specific facets and dynamic characteristics of complex 
concepts are not always obvious/ they are often difficult to 
teach and learn. For example/ research suggests that there are 
serious problems associated with learning 

disabilities classification decision making (Hofmeister/ 1983; 
Sabatino/ 1983; Thurlow & Ysseldyke/ 1979; Ysseldyke/ 1983). It 
appears that many of those charged with the responsibility of 
qualifying students as learning disabled have failed to 
accurately do so. 

Class LP. 2: An Expert System 

In response to the need for better systems for classifying 



133 

142 



LD.Train*r 
6 

LD Students F«rrara and Hofmttist«r (1984) d«v*loped an •xpmct 
ayafm, Class. LD2. Expert systems are programs which are 
designed to ask the computer user regarding a problem and provide 
the same advice cne might receive from an expert 
consultant (Harmon & King, 1984; Weiss & Kul ikowski , 1984). 
Class. LD2 is designed to provide the user with a second opinion 
regarding the probability that a student can be classified as LD. 
The class. LD2 knowledge base contains information and decision 
rules obtained from experts in the area of LD classification, 
federal law, state regulations, and current literature describing 
best practices in the area of student assessment (Ferrara, parry, 
6 Lubke, 1985). 

The knowledge base of an effective expert system must be 
designed to explicitly define each dimension and dynamic 
characteristic of the complex concept or concepts assocoated with 
its knowledge area. CLASS.LD2's rules define the concept 
•learning disabled student" (Martindale, Ferrara, & 
Campbell, 1986). 

LD. Trainer 

LD.Trainer (Prater « Althouse, 1986), a computer-assisted 
instruction (CAI) program, was developed to teach preaervice and 
inservice educators to make appropriate learning disabilities 
classification decisions. The program was designed to utilize 
the rules of CLASS. LD2 in a simple and coat-effective 
instructional format. Individuals who completed training with 
this system should be able to accurately discriminate between 
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students who should and should not bs classifisd as learning 
disabled in the State of Utah. 

LD.Trainer consists of a series of 13 lessons/ each of which 
present a portion of the concept "learning disabled student." 
The lessons teach preservice and inservice teachers to 
determining that a) the student manifests a discrepancy between 
expected and actual academic achievement; b) the student's 
learning difficulties are not due to other handicapping 
conditions (i.e./ vision/ hearing/ health problems); or c)the 
student's learning difficulties are not due to cultural/ 
economic/ or environmental factors. 

In developing the lessons for LD.Trainer/ a modified version 
of the concept instruction model suggested by Merrill and 
Tennyson (1977) was employed. Each lesson was divided into two 
parts/ instruction and practice. 

During instruction/ trainees are given a de.^inition which 
describes the important attributes in distinguishing an LD 
student and are then given matched examples and nor.fixamples of 
this concept. The trainees read a brief case study describing an 
examples of an LD student. Data from the case study are then fed 
into the computer and the tra inees examine the system' s 
conclusions regarding whether the student can qualify as LD. 
Following this/ they read an explanation of hov the system 
arrived at its conclusions. The explanation details how certain 
attributes (i.e./ those listed in the definition)/ were used to 
arrive at this conclusion. The trainees then read a second case 
study/ a nonexample/ and enter a second set of data. This time 
the value of the attributes of interest are changed and the 
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train^tts ac^ provided an opportunity! to view how changes in 
these attributes effect the system's conclusions. This process 
is repeated with different case studies but focusing on the same 
attribute. 

During the practice portion of each lesson the trainees 
again read a brief case study. After doing so, they make a 
decision regarding whether the student is LD and write the 
decision and justification for their decision in che printed 
material provided. The trainees then enter the data from the 
case study into the computer and compare their written 
conclusions with the system's. The printed materials provide 
justification for the system's conclusions by focusing on the 
same attribute as was presented and taught during the 
instructional part of the lesson. This process is repeated with 
a second case study. Finally, the trainees are given an 
opportunity to make LD classification decisions on case study 
information and then, varying the attribute of interest and 
holding all the other student data constant, the trainees can use 
the computer to check the accuracy of their decisions. 

Research 

As part of a week-long inservice program, twenty-one 
practicing teachers and administrators participated in one of two 
training conditions. Eleven completed LD.Trainer materials and 
ten were given representative special education student files and 
ran consultations with the expert system, CLASS. LD2. 
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On a tmat which served aa both the pre and the poatteat# 
traineea were given 12 caae atudiea on which they identified the 
atudent aa qualifying or not qualifying aa learning diaabled. 
Preliminary reaulta indicate that the traineea in both groupa 
improved their performance following participation in the 
training activitiea. However^ thoae traineea in the LD. Trainer 
group acored aignif icantly (p < .01) higher than the 
CLASS.LD2 group. 

Concluaiona 

Preliminary reaearch comparing CLASS.LD2 and LD.Trainer 
auggeata th^^t both ayatema are effective in teaching the complex 
concept, "learning diaabled atudent." Traineea who ran data from 
filea on learning diaabled atudenta through the CLASS.LD2 expert 
ayatem improved in their ability to accurately identify atudenta 
who could and could not be claaaified learning diaabled. 
However/ traineea who ayatema tically completed the LD.Trainer 
materiala ahowed even greater improvement in their ability to 
accurately identify learning diaabled atudenta. 

The aucceaa of the LD.Trainer program demonatratea the 
potential of modified expert ayatema aa tool a for teaching 
complex concepta. LD.Trainer ayatema tica 1 1 y variea each 
attribute which effect the three dimenaiona of the complex 
concept, "learning diaabled atudent" and allowa traineea to 
obaerve how changea in the attributea change the concluaiona 
drawn. The aaaence of the ayatem ia that relative to each 
attribute, traineea are preaented with examplea and nonexaroplea 
of the concept "learning diaabled atudent" that are wholly 
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dttpttnd«nt upon changes in tha valua of that attributa. This ia 
aaaantially tha procadura uaad in taaching ainpla concapc auch aa 
a color. Bowavar, LD.Tcainac aya tama tica 1 1 y appliaa tha 
procadura to aach attributa in aach dimanaion of tha complax 
concapt and providaa trainaaa with ?n opportunity to laarn how 
tha attributaa and diraanaiona intaract. 

LD.Trainar ia ona of tha firat CAl programa that haa 
attamptad to taach a conplax concapt. It haa damonatratad that 
affactiva inatructional programa can ba davalopad by combining 
axpart ayatam tachnology with what ia known about affactiva 
concapt inatruction. In so doing it opana tha door for tha 
davalopmant of othar programa that could afficiantly and coat- 
affactivaly taach othar complax concapta. 
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Figure Caption 
Fiqura 1 . concapt of laarning disabilitias. 
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Figura Caption 
Figuf 2 . LO with two unc«rtain dimensions. 
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